[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:01 dist-test-slave-vq2q krb5kdc[12334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:01 dist-test-slave-vq2q krb5kdc[12334](info): set up 2 sockets
Aug 12 02:29:01 dist-test-slave-vq2q krb5kdc[12334](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 02:29:02.762789 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:02 dist-test-slave-vq2q krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:03.011359 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43533
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:03.670732 12351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:03.671247 12351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:03.671417 12351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:03.681107 12351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:03.681339 12351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:03.681450 12351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:03.681612 12351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:03.681732 12351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:03.696966 12351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:43533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:03.699282 12351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:03.701366 12351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:03.727130 12357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:03.727169 12359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:03.729717 12361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:03.732569 12351 server_base.cc:1034] running on GCE node
I20260812 02:29:03.734043 12351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:03.736770 12351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:03.740236 12351 hybrid_clock.cc:648] HybridClock initialized: now 1786501743740023 us; error 138 us; skew 500 ppm
I20260812 02:29:03.740969 12351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:03 dist-test-slave-vq2q krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501743, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:03.746554 12351 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:03.750479 12351 webserver.cc:460] Webserver started at http://127.12.8.254:45903/ using document root <none> and password file <none>
I20260812 02:29:03.751874 12351 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:03.752110 12351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:03.752736 12351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:03.758309 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "0d28223617d04db1a36d801ab47d2ad1"
format_stamp: "Formatted at 2026-08-12 02:29:03 on dist-test-slave-vq2q"
server_key: "50900baac146821ee465bcb5d2b9c024"
server_key_iv: "08d20a48528ea6a98cde010f6d8d4d59"
server_key_version: "encryptionkey@0"
I20260812 02:29:03.759649 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "0d28223617d04db1a36d801ab47d2ad1"
format_stamp: "Formatted at 2026-08-12 02:29:03 on dist-test-slave-vq2q"
server_key: "50900baac146821ee465bcb5d2b9c024"
server_key_iv: "08d20a48528ea6a98cde010f6d8d4d59"
server_key_version: "encryptionkey@0"
I20260812 02:29:03.770697 12351 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.001s
I20260812 02:29:03.780092 12369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:03.783366 12351 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260812 02:29:03.783782 12351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "0d28223617d04db1a36d801ab47d2ad1"
format_stamp: "Formatted at 2026-08-12 02:29:03 on dist-test-slave-vq2q"
server_key: "50900baac146821ee465bcb5d2b9c024"
server_key_iv: "08d20a48528ea6a98cde010f6d8d4d59"
server_key_version: "encryptionkey@0"
I20260812 02:29:03.784163 12351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:03.804762 12351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:03.812750 12351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:03.813421 12351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:03.871188 12351 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:43533
I20260812 02:29:03.871626 12428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:43533 every 8 connection(s)
I20260812 02:29:03.874383 12351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:03.878962 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 12351
I20260812 02:29:03.879424 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:03.882355 12323 external_mini_cluster.cc:1428] Setting key 7aba2180eb6ca834ce4f969ff893ea0e
I20260812 02:29:03.884971 12430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:03.912941 12430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1: Bootstrap starting.
I20260812 02:29:03.923085 12430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:03.926946 12430 log.cc:824] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:29:03 dist-test-slave-vq2q krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:03.934671 12430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1: No bootstrap required, opened a new log
I20260812 02:29:03.947942 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:03.890432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.12.8.254:43533)
0812 02:29:03.892890 (+ 2458us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:03.892910 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:03.894847 (+ 1937us) server_negotiation.cc:407] Connection header received
0812 02:29:03.910794 (+ 15947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:03.910864 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:03.911595 (+ 731us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:03.912139 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:03.917125 (+ 4986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:03.919846 (+ 2721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:03.922680 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:03.924098 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:03.924505 (+ 407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:03.934604 (+ 10099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:03.934684 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:03.934728 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:03.934810 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:03.940387 (+ 5577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:03.941636 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:03.941643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:03.941649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:03.941959 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:03.942593 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:03.942597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:03.942599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:03.943500 (+ 901us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:03.944130 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:03.945353 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0812 02:29:03.945851 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1904,"thread_start_us":1332,"threads_started":1}
I20260812 02:29:03.952571 12430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28223617d04db1a36d801ab47d2ad1" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43533 } }
I20260812 02:29:03.953716 12430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:03.954020 12430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d28223617d04db1a36d801ab47d2ad1, State: Initialized, Role: FOLLOWER
I20260812 02:29:03.955946 12430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28223617d04db1a36d801ab47d2ad1" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43533 } }
I20260812 02:29:03.956588 12430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:03.957023 12430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:03.957379 12430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:03.961848 12430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28223617d04db1a36d801ab47d2ad1" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43533 } }
I20260812 02:29:03.963666 12430 leader_election.cc:302] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d28223617d04db1a36d801ab47d2ad1; no voters:
I20260812 02:29:03.965351 12430 leader_election.cc:288] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:03.965741 12437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:03.967468 12437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 0d28223617d04db1a36d801ab47d2ad1, State: Running, Role: LEADER
I20260812 02:29:03.969069 12437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28223617d04db1a36d801ab47d2ad1" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43533 } }
I20260812 02:29:03.971716 12430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:03.975685 12439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d28223617d04db1a36d801ab47d2ad1. Latest consensus state: current_term: 1 leader_uuid: "0d28223617d04db1a36d801ab47d2ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28223617d04db1a36d801ab47d2ad1" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43533 } } }
I20260812 02:29:03.976544 12439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:03.977671 12440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d28223617d04db1a36d801ab47d2ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d28223617d04db1a36d801ab47d2ad1" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43533 } } }
I20260812 02:29:03.978379 12440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:03.983770 12443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:03.999876 12443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:04.022181 12443 catalog_manager.cc:1347] Generated new cluster ID: cd26532d627f4f11b9faa577938cd16a
I20260812 02:29:04.022429 12443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:04.053136 12443 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:04.055348 12443 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:04.073544 12443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d28223617d04db1a36d801ab47d2ad1: Generated new TSK 0
I20260812 02:29:04.075095 12443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:04.155460 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43533
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:04.817830 12460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:04.818305 12460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:04.818492 12460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:04.827607 12460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:04.827812 12460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:04.828001 12460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:04.840687 12460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:04.842703 12460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:04.844628 12460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:04.864526 12467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:04.864713 12472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:04.864691 12468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:04.865839 12460 server_base.cc:1034] running on GCE node
I20260812 02:29:04.866834 12460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:04.868201 12460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:04.869451 12460 hybrid_clock.cc:648] HybridClock initialized: now 1786501744869359 us; error 84 us; skew 500 ppm
I20260812 02:29:04.869935 12460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:04 dist-test-slave-vq2q krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501744, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:04.875311 12460 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:04.877910 12460 webserver.cc:460] Webserver started at http://127.12.8.193:35383/ using document root <none> and password file <none>
I20260812 02:29:04.879062 12460 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:04.879237 12460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:04.879664 12460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:04.884960 12460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "852be162e142448fa9840137081902c7"
format_stamp: "Formatted at 2026-08-12 02:29:04 on dist-test-slave-vq2q"
server_key: "e4570f9d9d6eec3fe2c2898ff2ff21f8"
server_key_iv: "ee6740562e9e82d238990b8c52c9b25e"
server_key_version: "encryptionkey@0"
I20260812 02:29:04.886322 12460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "852be162e142448fa9840137081902c7"
format_stamp: "Formatted at 2026-08-12 02:29:04 on dist-test-slave-vq2q"
server_key: "e4570f9d9d6eec3fe2c2898ff2ff21f8"
server_key_iv: "ee6740562e9e82d238990b8c52c9b25e"
server_key_version: "encryptionkey@0"
I20260812 02:29:04.895208 12460 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260812 02:29:04.900308 12479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:04.902351 12460 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:29:04.902585 12460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "852be162e142448fa9840137081902c7"
format_stamp: "Formatted at 2026-08-12 02:29:04 on dist-test-slave-vq2q"
server_key: "e4570f9d9d6eec3fe2c2898ff2ff21f8"
server_key_iv: "ee6740562e9e82d238990b8c52c9b25e"
server_key_version: "encryptionkey@0"
I20260812 02:29:04.902915 12460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:04.922043 12460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:04.928257 12460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:04.928639 12460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:04.930256 12460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:04.932407 12460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:04.932518 12460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:04.932732 12460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:04.932821 12460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:04.978533 12460 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:35561
I20260812 02:29:04.978556 12614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:35561 every 8 connection(s)
I20260812 02:29:04.980507 12460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:04.982649 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 12460
I20260812 02:29:04.982949 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:04.983927 12323 external_mini_cluster.cc:1428] Setting key ce7d25b7b744c615c8e8a3a5d8d50bd2
Aug 12 02:29:04 dist-test-slave-vq2q krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501744, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
I20260812 02:29:05.015331 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:04.985290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:42245 (local address 127.12.8.254:43533)
0812 02:29:04.986056 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:04.986062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:04.987380 (+ 1318us) server_negotiation.cc:407] Connection header received
0812 02:29:04.990593 (+ 3213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:04.990599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:04.990689 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:04.990798 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:04.993649 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:04.994614 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:04.997079 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:04.997826 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:04.997987 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:05.006496 (+ 8509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:05.006508 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:05.006513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:05.006576 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:05.010757 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:05.011941 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:05.011946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:05.011949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:05.012122 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:05.012736 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:05.012740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:05.012742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:05.013080 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:05.013290 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:05.014643 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0812 02:29:05.014934 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":457,"threads_started":1}
I20260812 02:29:05.017623 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:04.986070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43533 (local address 127.12.8.193:42245)
0812 02:29:04.987152 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:04.987221 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:04.990146 (+ 2925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:04.991179 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:04.991195 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:04.992241 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:04.993392 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:04.993428 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:04.994818 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:04.994824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:04.996878 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:04.996897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:04.998120 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:04.998158 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:04.998420 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:05.002984 (+ 4564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:05.003024 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:05.006042 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:05.010984 (+ 4942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:05.010995 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:05.011022 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:05.011766 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:05.012248 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:05.012252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:05.012255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:05.012596 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:05.013295 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:05.013307 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:05.013931 (+ 624us) client_negotiation.cc:769] Sending connection context
0812 02:29:05.014515 (+ 584us) client_negotiation.cc:240] Negotiation successful
0812 02:29:05.015345 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":408,"threads_started":1}
I20260812 02:29:05.020185 12615 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43533
I20260812 02:29:05.020943 12615 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:05.022607 12615 heartbeater.cc:511] Master 127.12.8.254:43533 requested a full tablet report, sending...
I20260812 02:29:05.027598 12390 ts_manager.cc:194] Registered new tserver with Master: 852be162e142448fa9840137081902c7 (127.12.8.193:35561)
I20260812 02:29:05.030387 12390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:42245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:05.046911 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43533
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:05.667685 12621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:05.668097 12621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:05.668311 12621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:05.679319 12621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:05.679535 12621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:05.679782 12621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:05.694248 12621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:05.695978 12621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:05.698068 12621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:05.716735 12626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:05.717881 12630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:05.717944 12632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:05.719365 12621 server_base.cc:1034] running on GCE node
I20260812 02:29:05.720183 12621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:05.721606 12621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:05.722966 12621 hybrid_clock.cc:648] HybridClock initialized: now 1786501745722762 us; error 125 us; skew 500 ppm
I20260812 02:29:05.723488 12621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:05 dist-test-slave-vq2q krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501745, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:05.728128 12621 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:05.730232 12621 webserver.cc:460] Webserver started at http://127.12.8.194:43297/ using document root <none> and password file <none>
I20260812 02:29:05.731101 12621 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:05.731312 12621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:05.731782 12621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:05.736083 12621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "286e2ccff12247588c8c13de2e4667b0"
format_stamp: "Formatted at 2026-08-12 02:29:05 on dist-test-slave-vq2q"
server_key: "a3ddc7f066a46a9dce0c4e4c14caea9c"
server_key_iv: "c81065eabc74f8e4349d2c1efaa1e51c"
server_key_version: "encryptionkey@0"
I20260812 02:29:05.737080 12621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "286e2ccff12247588c8c13de2e4667b0"
format_stamp: "Formatted at 2026-08-12 02:29:05 on dist-test-slave-vq2q"
server_key: "a3ddc7f066a46a9dce0c4e4c14caea9c"
server_key_iv: "c81065eabc74f8e4349d2c1efaa1e51c"
server_key_version: "encryptionkey@0"
I20260812 02:29:05.745255 12621 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260812 02:29:05.751010 12641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:05.753057 12621 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260812 02:29:05.753391 12621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "286e2ccff12247588c8c13de2e4667b0"
format_stamp: "Formatted at 2026-08-12 02:29:05 on dist-test-slave-vq2q"
server_key: "a3ddc7f066a46a9dce0c4e4c14caea9c"
server_key_iv: "c81065eabc74f8e4349d2c1efaa1e51c"
server_key_version: "encryptionkey@0"
I20260812 02:29:05.753726 12621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:05.779894 12621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:05.785811 12621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:05.786140 12621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:05.787384 12621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:05.789631 12621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:05.789777 12621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:05.789914 12621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:05.789986 12621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:05.882607 12621 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:40733
I20260812 02:29:05.882742 12776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:40733 every 8 connection(s)
I20260812 02:29:05.884364 12621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:05.886016 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 12621
I20260812 02:29:05.886354 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:05.887117 12323 external_mini_cluster.cc:1428] Setting key 89f7edda4c8e40b7e42664663ee0c0b6
Aug 12 02:29:05 dist-test-slave-vq2q krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501745, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
I20260812 02:29:05.919561 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:05.888664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:52091 (local address 127.12.8.254:43533)
0812 02:29:05.889611 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:05.889616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:05.891387 (+ 1771us) server_negotiation.cc:407] Connection header received
0812 02:29:05.894637 (+ 3250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:05.894647 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:05.894778 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:05.894968 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:05.897465 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:05.898935 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:05.901396 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:05.902253 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:05.902386 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:05.909168 (+ 6782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:05.909210 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:05.909216 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:05.909272 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:05.914447 (+ 5175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:05.915964 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:05.915971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:05.915975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:05.916205 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:05.917008 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:05.917014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:05.917018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:05.917577 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:05.917766 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:05.918997 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0812 02:29:05.919207 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":521,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
I20260812 02:29:05.921310 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:05.889739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43533 (local address 127.12.8.194:52091)
0812 02:29:05.891158 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:05.891237 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:05.894268 (+ 3031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:05.895302 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:05.895311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:05.896159 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:05.897125 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:05.897148 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:05.899208 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:05.899212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:05.901168 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:05.901222 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:05.902643 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:05.902652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:05.903076 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:05.904285 (+ 1209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:05.904349 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:05.908611 (+ 4262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:05.914746 (+ 6135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:05.914760 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:05.914783 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:05.915749 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:05.916402 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:05.916407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:05.916410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:05.916843 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:05.917794 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:05.917802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:05.918420 (+ 618us) client_negotiation.cc:769] Sending connection context
0812 02:29:05.918832 (+ 412us) client_negotiation.cc:240] Negotiation successful
0812 02:29:05.919431 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":605,"threads_started":1}
I20260812 02:29:05.923687 12777 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43533
I20260812 02:29:05.924218 12777 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:05.925571 12777 heartbeater.cc:511] Master 127.12.8.254:43533 requested a full tablet report, sending...
I20260812 02:29:05.928110 12390 ts_manager.cc:194] Registered new tserver with Master: 286e2ccff12247588c8c13de2e4667b0 (127.12.8.194:40733)
I20260812 02:29:05.929507 12390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:52091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:05.953936 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43533
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:29:06.035588 12615 heartbeater.cc:503] Master 127.12.8.254:43533 was elected leader, sending a full tablet report...
W20260812 02:29:06.632623 12784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:06.633023 12784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:06.633299 12784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:06.644320 12784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:06.644562 12784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:06.644809 12784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:06.659040 12784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:06.660774 12784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:06.662719 12784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:06.684384 12792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:06.684914 12794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:06.685142 12796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:06.687292 12784 server_base.cc:1034] running on GCE node
I20260812 02:29:06.688189 12784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:06.689742 12784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:06.691053 12784 hybrid_clock.cc:648] HybridClock initialized: now 1786501746690950 us; error 63 us; skew 500 ppm
I20260812 02:29:06.691502 12784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:06 dist-test-slave-vq2q krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501746, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:06.699988 12784 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:29:06.703161 12784 webserver.cc:460] Webserver started at http://127.12.8.195:35807/ using document root <none> and password file <none>
I20260812 02:29:06.704437 12784 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:06.704681 12784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:06.705237 12784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:06.710328 12784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "e1103698b05143b2a531e2f5a37280dc"
format_stamp: "Formatted at 2026-08-12 02:29:06 on dist-test-slave-vq2q"
server_key: "6d543c7a18cd37d9aa7e192a05372c7f"
server_key_iv: "2a23289df3662dac71cd87a22bf672e5"
server_key_version: "encryptionkey@0"
I20260812 02:29:06.711513 12784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "e1103698b05143b2a531e2f5a37280dc"
format_stamp: "Formatted at 2026-08-12 02:29:06 on dist-test-slave-vq2q"
server_key: "6d543c7a18cd37d9aa7e192a05372c7f"
server_key_iv: "2a23289df3662dac71cd87a22bf672e5"
server_key_version: "encryptionkey@0"
I20260812 02:29:06.720815 12784 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.005s
I20260812 02:29:06.727331 12804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:06.729496 12784 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260812 02:29:06.729782 12784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "e1103698b05143b2a531e2f5a37280dc"
format_stamp: "Formatted at 2026-08-12 02:29:06 on dist-test-slave-vq2q"
server_key: "6d543c7a18cd37d9aa7e192a05372c7f"
server_key_iv: "2a23289df3662dac71cd87a22bf672e5"
server_key_version: "encryptionkey@0"
I20260812 02:29:06.730137 12784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:06.763424 12784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:06.770570 12784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:06.771121 12784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:06.773075 12784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:06.775655 12784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:06.775794 12784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:06.776018 12784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:06.776145 12784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:06.875710 12784 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:38837
I20260812 02:29:06.876030 12938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:38837 every 8 connection(s)
I20260812 02:29:06.877514 12784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:06.882653 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 12784
I20260812 02:29:06.882966 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:06.883849 12323 external_mini_cluster.cc:1428] Setting key 477e165032e71df3805433002f1d0655
Aug 12 02:29:06 dist-test-slave-vq2q krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501746, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:06.913666 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:06.884017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:60263 (local address 127.12.8.254:43533)
0812 02:29:06.885069 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:06.885076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:06.886774 (+ 1698us) server_negotiation.cc:407] Connection header received
0812 02:29:06.891124 (+ 4350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:06.891131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:06.891223 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:06.891332 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:06.893776 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:06.894862 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:06.897146 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:06.897931 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:06.898077 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:06.903818 (+ 5741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:06.903828 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:06.903835 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:06.903901 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:06.908812 (+ 4911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:06.910107 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:06.910112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:06.910115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:06.910437 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:06.911053 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:06.911057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:06.911059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:06.911529 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:06.911683 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:06.912723 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0812 02:29:06.912982 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"spinlock_wait_cycles":4352,"thread_start_us":882,"threads_started":1}
I20260812 02:29:06.914901 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:06.883529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43533 (local address 127.12.8.195:60263)
0812 02:29:06.886539 (+ 3010us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:06.886605 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:06.890716 (+ 4111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:06.891607 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:06.891618 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:06.892440 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:06.893525 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:06.893551 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:06.895043 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:06.895047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:06.896991 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:06.897005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:06.898300 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:06.898310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:06.898611 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:06.899676 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:06.899704 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:06.903438 (+ 3734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:06.909064 (+ 5626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:06.909074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:06.909092 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:06.909905 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:06.910596 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:06.910600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:06.910602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:06.910929 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:06.911704 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:06.911712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:06.912242 (+ 530us) client_negotiation.cc:769] Sending connection context
0812 02:29:06.912621 (+ 379us) client_negotiation.cc:240] Negotiation successful
0812 02:29:06.913327 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2538,"thread_start_us":2002,"threads_started":1}
I20260812 02:29:06.916962 12939 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43533
I20260812 02:29:06.917786 12939 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:06.919170 12939 heartbeater.cc:511] Master 127.12.8.254:43533 requested a full tablet report, sending...
I20260812 02:29:06.921892 12389 ts_manager.cc:194] Registered new tserver with Master: e1103698b05143b2a531e2f5a37280dc (127.12.8.195:38837)
I20260812 02:29:06.923453 12389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:60263
I20260812 02:29:06.931980 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:29:06.934507 12777 heartbeater.cc:503] Master 127.12.8.254:43533 was elected leader, sending a full tablet report...
Aug 12 02:29:06 dist-test-slave-vq2q krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:29:06 dist-test-slave-vq2q krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501746, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:07.009395 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:06.982012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.12.8.193:35561)
0812 02:29:06.982911 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:06.982923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:06.983053 (+ 130us) server_negotiation.cc:407] Connection header received
0812 02:29:06.983332 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:06.983342 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:06.983645 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:06.983828 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:06.986816 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:06.988263 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:06.990884 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:06.991759 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:06.991932 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:06.999794 (+ 7862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:06.999816 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:06.999834 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:06.999914 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:07.004788 (+ 4874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.006240 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.006249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.006251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.006519 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.007055 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.007059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.007061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.007584 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:07.008049 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:07.008316 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 02:29:07.008679 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":404,"threads_started":1}
W20260812 02:29:07.011338 12517 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53072
Aug 12 02:29:07 dist-test-slave-vq2q krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:29:07 dist-test-slave-vq2q krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:07.080265 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:07.061406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.12.8.193:35561)
0812 02:29:07.061685 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:07.061693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:07.061802 (+ 109us) server_negotiation.cc:407] Connection header received
0812 02:29:07.062093 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:07.062098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:07.062174 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:07.062265 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:07.063720 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:07.064705 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:07.066058 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:07.066507 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:07.066620 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:07.072656 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:07.072663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:07.072666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:07.072712 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:07.076763 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.077728 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.077733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.077735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.077946 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.078390 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.078394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.078396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.078723 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:07.079193 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:07.079524 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 02:29:07.079800 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 02:29:07.082821 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 12460
I20260812 02:29:07.101634 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:07.084133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.12.8.193:35561)
0812 02:29:07.084366 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:07.084371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:07.084744 (+ 373us) server_negotiation.cc:407] Connection header received
0812 02:29:07.084906 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:07.084911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:07.084974 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:07.085093 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:07.086524 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:07.087470 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:07.089168 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:07.089942 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:07.090050 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:07.093875 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:07.093882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:07.093886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:07.093924 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:07.097801 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.098891 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.098896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.098898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.099191 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.099764 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.099769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.099771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.100227 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:07.100495 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:07.100873 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 02:29:07.101122 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:29:07.102807 12609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:07.197499 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 12460
I20260812 02:29:07.219429 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 12621
Aug 12 02:29:07 dist-test-slave-vq2q krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:29:07.240504 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:07.220911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.12.8.194:40733)
0812 02:29:07.222004 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:07.222014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:07.222047 (+ 33us) server_negotiation.cc:407] Connection header received
0812 02:29:07.222132 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:07.222141 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:07.222471 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:07.222666 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:07.224210 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:07.225589 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:07.227064 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:07.227769 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:07.227908 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:07.233278 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:07.233300 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:07.233321 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:07.233375 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:07.236848 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.237919 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.237928 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.237931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.238092 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.238570 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.238574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.238577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.239030 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:07.239376 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:07.239759 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 02:29:07.240089 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":615,"threads_started":1}
I20260812 02:29:07.242165 12772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:07.333148 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 12621
I20260812 02:29:07.355762 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 12784
Aug 12 02:29:07 dist-test-slave-vq2q krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:29:07.374727 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:07.357059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.12.8.195:38837)
0812 02:29:07.357743 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:07.357751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:07.357787 (+ 36us) server_negotiation.cc:407] Connection header received
0812 02:29:07.357868 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:07.357875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:07.358166 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:07.358346 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:07.359835 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:07.360937 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:07.362684 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:07.363351 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:07.363483 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:07.368219 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:07.368234 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:07.368254 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:07.368299 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:07.371342 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.372145 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.372154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.372157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.372306 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:07.372727 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:07.372731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:07.372733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:07.373300 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:07.373682 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:07.373964 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 02:29:07.374298 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":309,"threads_started":1}
I20260812 02:29:07.376447 12934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:07.463208 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 12784
I20260812 02:29:07.488087 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 12351
I20260812 02:29:07.489574 12424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:07.609503 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 12351
2026-08-12T02:29:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6059 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:07 dist-test-slave-vq2q krb5kdc[12965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:07 dist-test-slave-vq2q krb5kdc[12965](info): set up 2 sockets
Aug 12 02:29:07 dist-test-slave-vq2q krb5kdc[12965](info): commencing operation
krb5kdc: starting...
W20260812 02:29:08.738234 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:08 dist-test-slave-vq2q krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:08.989506 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45453
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:45453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:09.400353 12981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:09.400797 12981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:09.400938 12981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:09.409291 12981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:09.409463 12981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:09.409541 12981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:09.409646 12981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:09.409715 12981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:09.421348 12981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:45453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:09.423202 12981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:09.424865 12981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:09.439246 12987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:09.439177 12990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:09.439306 12988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:09.440670 12981 server_base.cc:1034] running on GCE node
I20260812 02:29:09.441707 12981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:09.443506 12981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:09.444736 12981 hybrid_clock.cc:648] HybridClock initialized: now 1786501749444666 us; error 69 us; skew 500 ppm
I20260812 02:29:09.445127 12981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:09 dist-test-slave-vq2q krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501749, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:09.448864 12981 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:09.450789 12981 webserver.cc:460] Webserver started at http://127.12.8.254:34081/ using document root <none> and password file <none>
I20260812 02:29:09.451670 12981 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:09.451802 12981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:09.452108 12981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:09.455567 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a"
format_stamp: "Formatted at 2026-08-12 02:29:09 on dist-test-slave-vq2q"
server_key: "cda468d0851c18b041dd51a68217b40f"
server_key_iv: "0569a9be8f6863c4b927de1227073cd2"
server_key_version: "encryptionkey@0"
I20260812 02:29:09.456331 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a"
format_stamp: "Formatted at 2026-08-12 02:29:09 on dist-test-slave-vq2q"
server_key: "cda468d0851c18b041dd51a68217b40f"
server_key_iv: "0569a9be8f6863c4b927de1227073cd2"
server_key_version: "encryptionkey@0"
I20260812 02:29:09.462733 12981 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 02:29:09.466810 13001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:09.468715 12981 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:09.468936 12981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a"
format_stamp: "Formatted at 2026-08-12 02:29:09 on dist-test-slave-vq2q"
server_key: "cda468d0851c18b041dd51a68217b40f"
server_key_iv: "0569a9be8f6863c4b927de1227073cd2"
server_key_version: "encryptionkey@0"
I20260812 02:29:09.469146 12981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:09.492259 12981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:09.497387 12981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:09.497674 12981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:09.518442 12981 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:45453
I20260812 02:29:09.518567 13063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:45453 every 8 connection(s)
I20260812 02:29:09.520004 12981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:09.525322 13064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:09.526384 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 12981
I20260812 02:29:09.526684 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:09.527647 12323 external_mini_cluster.cc:1428] Setting key e78e42faaf36329a6bf77b8ca83d9e25
I20260812 02:29:09.544014 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a: Bootstrap starting.
Aug 12 02:29:09 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:09.548265 13064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:09.549597 13064 log.cc:824] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:09.553747 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a: No bootstrap required, opened a new log
I20260812 02:29:09.558321 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:09.530370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.12.8.254:45453)
0812 02:29:09.531398 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:09.531415 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:09.531466 (+ 51us) server_negotiation.cc:407] Connection header received
0812 02:29:09.533931 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:09.533962 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:09.534446 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:09.534860 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:09.537138 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:09.538554 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:09.540608 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:09.541463 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:09.541637 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:09.547491 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:09.547503 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:09.547529 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:09.547589 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:09.552291 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:09.553546 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:09.553555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:09.553563 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:09.553804 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:09.554418 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:09.554424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:09.554429 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:09.555178 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:09.555589 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:09.556161 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 02:29:09.556757 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":496,"threads_started":1}
I20260812 02:29:09.562824 13064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45453 } }
I20260812 02:29:09.563275 13064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:09.563371 13064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e75d3977c054b2f8b03e3b86dbbdc2a, State: Initialized, Role: FOLLOWER
I20260812 02:29:09.564016 13064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45453 } }
I20260812 02:29:09.564359 13064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:09.564535 13064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:09.564710 13064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:09.567157 13064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45453 } }
I20260812 02:29:09.567694 13064 leader_election.cc:302] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e75d3977c054b2f8b03e3b86dbbdc2a; no voters:
I20260812 02:29:09.568368 13064 leader_election.cc:288] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:09.568645 13069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:09.569581 13069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [term 1 LEADER]: Becoming Leader. State: Replica: 9e75d3977c054b2f8b03e3b86dbbdc2a, State: Running, Role: LEADER
I20260812 02:29:09.570338 13069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45453 } }
I20260812 02:29:09.570761 13064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:09.574229 13070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45453 } } }
I20260812 02:29:09.574659 13070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:09.575670 13080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:09.575428 13071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e75d3977c054b2f8b03e3b86dbbdc2a. Latest consensus state: current_term: 1 leader_uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e75d3977c054b2f8b03e3b86dbbdc2a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45453 } } }
I20260812 02:29:09.575846 13071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:09.584079 13080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:09.595937 13080 catalog_manager.cc:1347] Generated new cluster ID: cebe7d740a8040b7a6a749dce6427ee4
I20260812 02:29:09.596148 13080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:09.609256 13080 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:09.611522 13080 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:09.652809 13080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e75d3977c054b2f8b03e3b86dbbdc2a: Generated new TSK 0
I20260812 02:29:09.654202 13080 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:09.731076 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45453
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:10.162921 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:10.163311 13096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:10.163455 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:10.171653 13096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:10.171823 13096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:10.172016 13096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:10.183579 13096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:10.185506 13096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:10.187372 13096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:10.205049 13101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:10.205049 13102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:10.205435 13107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:10.206305 13096 server_base.cc:1034] running on GCE node
I20260812 02:29:10.207185 13096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:10.208488 13096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:10.209781 13096 hybrid_clock.cc:648] HybridClock initialized: now 1786501750209713 us; error 41 us; skew 500 ppm
I20260812 02:29:10.210229 13096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:10 dist-test-slave-vq2q krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:10.214195 13096 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:10.216038 13096 webserver.cc:460] Webserver started at http://127.12.8.193:32849/ using document root <none> and password file <none>
I20260812 02:29:10.216825 13096 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:10.216951 13096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:10.217319 13096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:10.220593 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "fadcf3359c444a3394de3d8acad8ca89"
format_stamp: "Formatted at 2026-08-12 02:29:10 on dist-test-slave-vq2q"
server_key: "1cbcd415043631e11e25a2a7c95380e6"
server_key_iv: "8a2f28b2a85d5385dff82bea97a1190a"
server_key_version: "encryptionkey@0"
I20260812 02:29:10.221479 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "fadcf3359c444a3394de3d8acad8ca89"
format_stamp: "Formatted at 2026-08-12 02:29:10 on dist-test-slave-vq2q"
server_key: "1cbcd415043631e11e25a2a7c95380e6"
server_key_iv: "8a2f28b2a85d5385dff82bea97a1190a"
server_key_version: "encryptionkey@0"
I20260812 02:29:10.227419 13096 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260812 02:29:10.231580 13118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:10.233328 13096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:29:10.233510 13096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "fadcf3359c444a3394de3d8acad8ca89"
format_stamp: "Formatted at 2026-08-12 02:29:10 on dist-test-slave-vq2q"
server_key: "1cbcd415043631e11e25a2a7c95380e6"
server_key_iv: "8a2f28b2a85d5385dff82bea97a1190a"
server_key_version: "encryptionkey@0"
I20260812 02:29:10.233706 13096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:10.258363 13096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:10.262918 13096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:10.263197 13096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:10.264323 13096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:10.265898 13096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:10.265980 13096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:10.266104 13096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:10.266162 13096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:10.303543 13096 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:33995
I20260812 02:29:10.303905 13254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:33995 every 8 connection(s)
I20260812 02:29:10.304975 13096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:10.311650 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 13096
I20260812 02:29:10.311923 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:10.312804 12323 external_mini_cluster.cc:1428] Setting key 3696fe3f2e1c1bcb340f888de379aacc
Aug 12 02:29:10 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:10.332746 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:10.308480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:50559 (local address 127.12.8.254:45453)
0812 02:29:10.309061 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:10.309066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:10.310254 (+ 1188us) server_negotiation.cc:407] Connection header received
0812 02:29:10.313314 (+ 3060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:10.313320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:10.313410 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:10.313532 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:10.315870 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:10.316655 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:10.319297 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:10.319718 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:10.319825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:10.324843 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:10.324849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:10.324854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:10.324932 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:10.328437 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:10.329736 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:10.329742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:10.329745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:10.329952 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:10.330523 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:10.330528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:10.330531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:10.330892 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:10.331050 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:10.332098 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0812 02:29:10.332371 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":280,"threads_started":1}
I20260812 02:29:10.334105 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:10.309027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45453 (local address 127.12.8.193:50559)
0812 02:29:10.309993 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:10.310081 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:10.312816 (+ 2735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:10.313862 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:10.313874 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:10.314683 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:10.315651 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:10.315684 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:10.316842 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:10.316848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:10.319081 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:10.319102 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:10.320006 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:10.320013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:10.320370 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:10.321290 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:10.321323 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:10.324528 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:10.328693 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:10.328705 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:10.328726 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:10.329532 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:10.330105 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:10.330109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:10.330111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:10.330374 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:10.331113 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:10.331121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:10.331600 (+ 479us) client_negotiation.cc:769] Sending connection context
0812 02:29:10.331920 (+ 320us) client_negotiation.cc:240] Negotiation successful
0812 02:29:10.332492 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":395,"threads_started":1}
I20260812 02:29:10.336270 13256 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45453
I20260812 02:29:10.336664 13256 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:10.337705 13256 heartbeater.cc:511] Master 127.12.8.254:45453 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
I20260812 02:29:10.340359 13020 ts_manager.cc:194] Registered new tserver with Master: fadcf3359c444a3394de3d8acad8ca89 (127.12.8.193:33995)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
I20260812 02:29:10.342041 13020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:50559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:10.368978 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45453
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:10.739629 13263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:10.739923 13263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:10.740041 13263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:10.747416 13263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:10.747540 13263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:10.747684 13263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:10.757834 13263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:10.759264 13263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:10.760663 13263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:10.775991 13269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:10.776039 13268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:10.775941 13271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:10.777160 13263 server_base.cc:1034] running on GCE node
I20260812 02:29:10.777855 13263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:10.778973 13263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:10.780212 13263 hybrid_clock.cc:648] HybridClock initialized: now 1786501750780155 us; error 52 us; skew 500 ppm
I20260812 02:29:10.780560 13263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:10 dist-test-slave-vq2q krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:10.784046 13263 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:10.785792 13263 webserver.cc:460] Webserver started at http://127.12.8.194:43599/ using document root <none> and password file <none>
I20260812 02:29:10.786511 13263 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:10.786620 13263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:10.786947 13263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:10.790066 13263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "04caee0064914ecca0f8ddb4a42ea1eb"
format_stamp: "Formatted at 2026-08-12 02:29:10 on dist-test-slave-vq2q"
server_key: "41ce40db63ce2349dc6b2b1b9aa38ee8"
server_key_iv: "f86a76684439b2101ba0141333099000"
server_key_version: "encryptionkey@0"
I20260812 02:29:10.790948 13263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "04caee0064914ecca0f8ddb4a42ea1eb"
format_stamp: "Formatted at 2026-08-12 02:29:10 on dist-test-slave-vq2q"
server_key: "41ce40db63ce2349dc6b2b1b9aa38ee8"
server_key_iv: "f86a76684439b2101ba0141333099000"
server_key_version: "encryptionkey@0"
I20260812 02:29:10.796732 13263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:29:10.800918 13279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:10.802613 13263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:29:10.802795 13263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "04caee0064914ecca0f8ddb4a42ea1eb"
format_stamp: "Formatted at 2026-08-12 02:29:10 on dist-test-slave-vq2q"
server_key: "41ce40db63ce2349dc6b2b1b9aa38ee8"
server_key_iv: "f86a76684439b2101ba0141333099000"
server_key_version: "encryptionkey@0"
I20260812 02:29:10.803021 13263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:10.826825 13263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:10.831509 13263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:10.831808 13263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:10.832978 13263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:10.834643 13263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:10.834731 13263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:10.834825 13263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:10.834875 13263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:10.872916 13263 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:36323
I20260812 02:29:10.872948 13417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:36323 every 8 connection(s)
I20260812 02:29:10.874380 13263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:10.876391 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 13263
I20260812 02:29:10.876591 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:10.877079 12323 external_mini_cluster.cc:1428] Setting key 6be46af149e40963f6410131b089a4c2
Aug 12 02:29:10 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
I20260812 02:29:10.902137 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:10.878067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:47265 (local address 127.12.8.254:45453)
0812 02:29:10.878709 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:10.878715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:10.880303 (+ 1588us) server_negotiation.cc:407] Connection header received
0812 02:29:10.882960 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:10.882968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:10.883109 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:10.883252 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:10.886224 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:10.887091 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:10.888994 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:10.889570 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:10.889716 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:10.894366 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:10.894374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:10.894378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:10.894429 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:10.897740 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:10.898744 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:10.898749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:10.898752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:10.898964 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:10.899738 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:10.899742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:10.899745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:10.900235 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:10.900437 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:10.901367 (+ 930us) server_negotiation.cc:299] Negotiation successful
0812 02:29:10.901682 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":267,"threads_started":1}
I20260812 02:29:10.903060 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:10.878807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45453 (local address 127.12.8.194:47265)
0812 02:29:10.880042 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:10.880123 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:10.882460 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:10.883716 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:10.883733 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:10.884786 (+ 1053us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:10.885918 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:10.885953 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:10.887305 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:10.887309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:10.888794 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:10.888811 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:10.889881 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:10.889890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:10.890191 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:10.891064 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:10.891095 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:10.893969 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:10.897960 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:10.897976 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:10.897996 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:10.898587 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:10.899190 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:10.899197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:10.899200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:10.899545 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:10.900395 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:10.900403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:10.900832 (+ 429us) client_negotiation.cc:769] Sending connection context
0812 02:29:10.901136 (+ 304us) client_negotiation.cc:240] Negotiation successful
0812 02:29:10.901817 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":439,"threads_started":1}
I20260812 02:29:10.904868 13418 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45453
I20260812 02:29:10.905454 13418 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:10.906705 13418 heartbeater.cc:511] Master 127.12.8.254:45453 requested a full tablet report, sending...
I20260812 02:29:10.908926 13020 ts_manager.cc:194] Registered new tserver with Master: 04caee0064914ecca0f8ddb4a42ea1eb (127.12.8.194:36323)
I20260812 02:29:10.909861 13020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:47265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:10.933138 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45453
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:11.275537 13425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:11.275835 13425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:11.275957 13425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:11.283331 13425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:11.283456 13425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:11.283604 13425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:11.293613 13425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45453
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:11.295068 13425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:11.296402 13425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:11.311779 13432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:11.311920 13436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:11.312347 13431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:11.312565 13425 server_base.cc:1034] running on GCE node
I20260812 02:29:11.313395 13425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:11.314626 13425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:11.315891 13425 hybrid_clock.cc:648] HybridClock initialized: now 1786501751315826 us; error 47 us; skew 500 ppm
I20260812 02:29:11.316327 13425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:11 dist-test-slave-vq2q krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501751, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:11.320128 13425 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:29:11.321956 13425 webserver.cc:460] Webserver started at http://127.12.8.195:32813/ using document root <none> and password file <none>
I20260812 02:29:11.322832 13425 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:11.322969 13425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:11.323333 13425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:11.326987 13425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "ad25e125bbad4326946ac055a19946bc"
format_stamp: "Formatted at 2026-08-12 02:29:11 on dist-test-slave-vq2q"
server_key: "01be13dfab98be907e2444de4aac1fba"
server_key_iv: "aa3d523c3cbdaaa18a567f0261508ed4"
server_key_version: "encryptionkey@0"
I20260812 02:29:11.327864 13425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "ad25e125bbad4326946ac055a19946bc"
format_stamp: "Formatted at 2026-08-12 02:29:11 on dist-test-slave-vq2q"
server_key: "01be13dfab98be907e2444de4aac1fba"
server_key_iv: "aa3d523c3cbdaaa18a567f0261508ed4"
server_key_version: "encryptionkey@0"
I20260812 02:29:11.334385 13425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 02:29:11.338939 13445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:11.340787 13425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:29:11.341029 13425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "ad25e125bbad4326946ac055a19946bc"
format_stamp: "Formatted at 2026-08-12 02:29:11 on dist-test-slave-vq2q"
server_key: "01be13dfab98be907e2444de4aac1fba"
server_key_iv: "aa3d523c3cbdaaa18a567f0261508ed4"
server_key_version: "encryptionkey@0"
I20260812 02:29:11.341380 13425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:11.346503 13256 heartbeater.cc:503] Master 127.12.8.254:45453 was elected leader, sending a full tablet report...
I20260812 02:29:11.374330 13425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:11.379832 13425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:11.380193 13425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:11.381467 13425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:11.383150 13425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:11.383239 13425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:11.383375 13425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:11.383445 13425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:11.422510 13425 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:39475
I20260812 02:29:11.422658 13580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:39475 every 8 connection(s)
I20260812 02:29:11.424554 13425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:11.430567 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 13425
I20260812 02:29:11.430920 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:11.431651 12323 external_mini_cluster.cc:1428] Setting key 2b9439f581b294ba540e6ef460863590
Aug 12 02:29:11 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501751, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:11.455771 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.428642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:47759 (local address 127.12.8.254:45453)
0812 02:29:11.429353 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.429358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.431547 (+ 2189us) server_negotiation.cc:407] Connection header received
0812 02:29:11.435115 (+ 3568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.435121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.435217 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.435320 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:11.437686 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.438951 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.441090 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.441758 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.441879 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.447276 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:11.447285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:11.447290 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:11.447353 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:11.451689 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.452708 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.452712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.452715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.452905 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.453568 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.453571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.453573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.453945 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:11.454108 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.455072 (+ 964us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.455264 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":409,"threads_started":1}
I20260812 02:29:11.457404 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.429598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45453 (local address 127.12.8.195:47759)
0812 02:29:11.431344 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:11.431395 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:11.434816 (+ 3421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:11.435702 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:11.435716 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:11.436477 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:11.437487 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.437517 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.439136 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.439140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.440948 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.440961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.442045 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.442054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.442387 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.443214 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:11.443240 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:11.446895 (+ 3655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:11.451884 (+ 4989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:11.451895 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:11.451915 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:11.452575 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:11.453004 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:11.453008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:11.453010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:11.453456 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:11.454067 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:11.454073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:11.454598 (+ 525us) client_negotiation.cc:769] Sending connection context
0812 02:29:11.454907 (+ 309us) client_negotiation.cc:240] Negotiation successful
0812 02:29:11.455488 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":966,"threads_started":1}
I20260812 02:29:11.459513 13581 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45453
I20260812 02:29:11.460098 13581 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:11.461802 13581 heartbeater.cc:511] Master 127.12.8.254:45453 requested a full tablet report, sending...
I20260812 02:29:11.463881 13020 ts_manager.cc:194] Registered new tserver with Master: ad25e125bbad4326946ac055a19946bc (127.12.8.195:39475)
I20260812 02:29:11.464996 13020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:47759
I20260812 02:29:11.476248 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:29:11.507606 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.486640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.12.8.254:45453)
0812 02:29:11.486929 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.486935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.487954 (+ 1019us) server_negotiation.cc:407] Connection header received
0812 02:29:11.488117 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.488123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.488221 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.488349 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:11.491178 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.492251 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.494315 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.495085 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.495241 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.498905 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:11.498913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:11.498917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:11.498970 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:11.503358 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.504477 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.504485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.504488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.504765 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.505409 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.505414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.505416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.506033 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:11.506316 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.506800 (+ 484us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.507103 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 02:29:11.517428 13020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:11.531093 13020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:11.577569 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.554681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:33995 (local address 127.0.0.1:35628)
0812 02:29:11.561308 (+ 6627us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:11.561368 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:11.562538 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:11.564192 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:11.564200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:11.564239 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:11.565854 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.565869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.569160 (+ 3291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.569167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.573096 (+ 3929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.573109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.576141 (+ 3032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.576150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.576378 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.576441 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 02:29:11.576586 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 02:29:11.576865 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6282,"mutex_wait_us":38,"thread_start_us":5545,"threads_started":1}
I20260812 02:29:11.577809 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.555437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.12.8.193:33995)
0812 02:29:11.560619 (+ 5182us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.560631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.561527 (+ 896us) server_negotiation.cc:407] Connection header received
0812 02:29:11.562985 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.562997 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.563319 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.563596 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:11.566107 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.568871 (+ 2764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.573398 (+ 4527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.575460 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.575614 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.575920 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.576801 (+ 881us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.577074 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4911,"thread_start_us":4316,"threads_started":1}
I20260812 02:29:11.578971 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.560008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:39475 (local address 127.0.0.1:58150)
0812 02:29:11.565544 (+ 5536us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:11.565567 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:11.565928 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:11.567221 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:11.567225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:11.567248 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:11.569035 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.569054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.572027 (+ 2973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.572033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.575139 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.575158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.577934 (+ 2776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.577942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.578182 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.578210 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:29:11.578335 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 02:29:11.578474 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5326,"mutex_wait_us":30,"thread_start_us":4631,"threads_started":1}
I20260812 02:29:11.579224 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.555831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.12.8.195:39475)
0812 02:29:11.559231 (+ 3400us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.559241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.565696 (+ 6455us) server_negotiation.cc:407] Connection header received
0812 02:29:11.566150 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.566157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.566385 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.566536 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:11.569467 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.571806 (+ 2339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.575339 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.577228 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.577412 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.577712 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.578409 (+ 697us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.578593 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3091,"thread_start_us":2775,"threads_started":1}
I20260812 02:29:11.584193 13176 tablet_service.cc:1467] Processing CreateTablet for tablet f4c3f7cc3faa4a149c965f120a0f1cfe (DEFAULT_TABLE table=test-table [id=e24190ee889942be8476f004e396e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:11.584137 13506 tablet_service.cc:1467] Processing CreateTablet for tablet f4c3f7cc3faa4a149c965f120a0f1cfe (DEFAULT_TABLE table=test-table [id=e24190ee889942be8476f004e396e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:11.586345 13506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c3f7cc3faa4a149c965f120a0f1cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:11.587112 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.556272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.12.8.194:36323)
0812 02:29:11.557939 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.557952 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.565644 (+ 7692us) server_negotiation.cc:407] Connection header received
0812 02:29:11.565964 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.565971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.566342 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.566704 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:11.569522 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.572792 (+ 3270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.576230 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.585602 (+ 9372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.585774 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.586051 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.586265 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.586506 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":1036,"threads_started":1}
I20260812 02:29:11.588217 13176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c3f7cc3faa4a149c965f120a0f1cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:11.588433 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.559072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:36323 (local address 127.0.0.1:51530)
0812 02:29:11.565431 (+ 6359us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:11.565453 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:11.565692 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:11.566648 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:11.566655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:11.566712 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:11.569074 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.569085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.573078 (+ 3993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.573084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.575921 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.575936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.585873 (+ 9937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.585880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.586072 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.586107 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 02:29:11.586185 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 02:29:11.586358 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6180,"mutex_wait_us":4,"thread_start_us":5638,"threads_started":1}
I20260812 02:29:11.592924 13338 tablet_service.cc:1467] Processing CreateTablet for tablet f4c3f7cc3faa4a149c965f120a0f1cfe (DEFAULT_TABLE table=test-table [id=e24190ee889942be8476f004e396e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:11.595772 13338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c3f7cc3faa4a149c965f120a0f1cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:11.610090 13605 tablet_bootstrap.cc:492] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc: Bootstrap starting.
I20260812 02:29:11.616355 13606 tablet_bootstrap.cc:492] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89: Bootstrap starting.
I20260812 02:29:11.617972 13605 tablet_bootstrap.cc:654] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:11.619442 13607 tablet_bootstrap.cc:492] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb: Bootstrap starting.
I20260812 02:29:11.621590 13605 log.cc:824] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:11.624228 13606 tablet_bootstrap.cc:654] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:11.626047 13607 tablet_bootstrap.cc:654] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:11.626842 13605 tablet_bootstrap.cc:492] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc: No bootstrap required, opened a new log
I20260812 02:29:11.626921 13606 log.cc:824] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:11.627290 13605 ts_tablet_manager.cc:1397] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20260812 02:29:11.628525 13607 log.cc:824] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:11.632305 13606 tablet_bootstrap.cc:492] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89: No bootstrap required, opened a new log
I20260812 02:29:11.632802 13606 ts_tablet_manager.cc:1397] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260812 02:29:11.639842 13607 tablet_bootstrap.cc:492] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb: No bootstrap required, opened a new log
I20260812 02:29:11.640276 13607 ts_tablet_manager.cc:1397] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20260812 02:29:11.644935 13605 raft_consensus.cc:348] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.645812 13605 raft_consensus.cc:374] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:11.645998 13605 raft_consensus.cc:729] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad25e125bbad4326946ac055a19946bc, State: Initialized, Role: FOLLOWER
I20260812 02:29:11.647042 13605 consensus_queue.cc:260] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.649580 13581 heartbeater.cc:503] Master 127.12.8.254:45453 was elected leader, sending a full tablet report...
I20260812 02:29:11.650501 13605 ts_tablet_manager.cc:1428] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc: Time spent starting tablet: real 0.023s user 0.016s sys 0.008s
I20260812 02:29:11.650179 13606 raft_consensus.cc:348] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.650918 13606 raft_consensus.cc:374] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:11.651103 13606 raft_consensus.cc:729] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fadcf3359c444a3394de3d8acad8ca89, State: Initialized, Role: FOLLOWER
I20260812 02:29:11.652165 13606 consensus_queue.cc:260] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.652159 13607 raft_consensus.cc:348] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.652741 13607 raft_consensus.cc:374] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:11.652925 13607 raft_consensus.cc:729] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04caee0064914ecca0f8ddb4a42ea1eb, State: Initialized, Role: FOLLOWER
I20260812 02:29:11.654116 13607 consensus_queue.cc:260] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.655201 13606 ts_tablet_manager.cc:1428] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89: Time spent starting tablet: real 0.022s user 0.013s sys 0.011s
I20260812 02:29:11.657162 13418 heartbeater.cc:503] Master 127.12.8.254:45453 was elected leader, sending a full tablet report...
I20260812 02:29:11.657897 13607 ts_tablet_manager.cc:1428] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260812 02:29:11.661428 13615 raft_consensus.cc:482] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:11.661888 13615 raft_consensus.cc:504] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.664676 13615 leader_election.cc:288] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad25e125bbad4326946ac055a19946bc (127.12.8.195:39475), 04caee0064914ecca0f8ddb4a42ea1eb (127.12.8.194:36323)
I20260812 02:29:11.677722 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.665441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:46563 (local address 127.12.8.195:39475)
0812 02:29:11.665765 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.665770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.667701 (+ 1931us) server_negotiation.cc:407] Connection header received
0812 02:29:11.667890 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.667898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.668016 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.668184 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:11.669322 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.670977 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.674817 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.676280 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.676474 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.676549 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.677170 (+ 621us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.677386 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260812 02:29:11.677798 13582 tablet.cc:2369] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:11.678401 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.665448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:39475 (local address 127.12.8.193:46563)
0812 02:29:11.667519 (+ 2071us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:11.667536 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:11.667711 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:11.668237 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:11.668241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:11.668271 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:11.669058 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.669072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.671209 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.671214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.674516 (+ 3302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.674530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.676697 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.676706 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.676920 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.676960 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 02:29:11.677089 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 02:29:11.677286 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"thread_start_us":1604,"threads_started":1}
I20260812 02:29:11.680769 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.665749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:60951 (local address 127.12.8.194:36323)
0812 02:29:11.666041 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.666047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.671307 (+ 5260us) server_negotiation.cc:407] Connection header received
0812 02:29:11.671591 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.671598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.671699 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.671800 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:11.673227 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.674488 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.677489 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.679314 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.679458 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.679517 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.680296 (+ 779us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.680471 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 02:29:11.680904 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.666240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:36323 (local address 127.12.8.193:60951)
0812 02:29:11.671104 (+ 4864us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:11.671125 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:11.671345 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:11.671850 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:11.671857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:11.671887 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:11.672821 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.672837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.674694 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.674700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.677169 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:11.677227 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.679836 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.679844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:11.680066 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.680098 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:29:11.680209 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 02:29:11.680354 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4688,"thread_start_us":4407,"threads_started":1}
I20260812 02:29:11.680954 13527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c3f7cc3faa4a149c965f120a0f1cfe" candidate_uuid: "fadcf3359c444a3394de3d8acad8ca89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad25e125bbad4326946ac055a19946bc" is_pre_election: true
I20260812 02:29:11.682078 13527 raft_consensus.cc:2454] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fadcf3359c444a3394de3d8acad8ca89 in term 0.
I20260812 02:29:11.682440 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c3f7cc3faa4a149c965f120a0f1cfe" candidate_uuid: "fadcf3359c444a3394de3d8acad8ca89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" is_pre_election: true
I20260812 02:29:11.683264 13122 leader_election.cc:302] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad25e125bbad4326946ac055a19946bc, fadcf3359c444a3394de3d8acad8ca89; no voters:
I20260812 02:29:11.683249 13363 raft_consensus.cc:2454] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fadcf3359c444a3394de3d8acad8ca89 in term 0.
I20260812 02:29:11.683763 13615 raft_consensus.cc:2781] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:11.683965 13615 raft_consensus.cc:482] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:11.684149 13615 raft_consensus.cc:3037] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:11.688120 13615 raft_consensus.cc:504] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.689080 13615 leader_election.cc:288] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [CANDIDATE]: Term 1 election: Requested vote from peers ad25e125bbad4326946ac055a19946bc (127.12.8.195:39475), 04caee0064914ecca0f8ddb4a42ea1eb (127.12.8.194:36323)
I20260812 02:29:11.689831 13527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c3f7cc3faa4a149c965f120a0f1cfe" candidate_uuid: "fadcf3359c444a3394de3d8acad8ca89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad25e125bbad4326946ac055a19946bc"
I20260812 02:29:11.689932 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c3f7cc3faa4a149c965f120a0f1cfe" candidate_uuid: "fadcf3359c444a3394de3d8acad8ca89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04caee0064914ecca0f8ddb4a42ea1eb"
I20260812 02:29:11.690091 13527 raft_consensus.cc:3037] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:11.690349 13363 raft_consensus.cc:3037] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:11.693867 13527 raft_consensus.cc:2454] T f4c3f7cc3faa4a149c965f120a0f1cfe P ad25e125bbad4326946ac055a19946bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fadcf3359c444a3394de3d8acad8ca89 in term 1.
I20260812 02:29:11.694494 13122 leader_election.cc:302] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad25e125bbad4326946ac055a19946bc, fadcf3359c444a3394de3d8acad8ca89; no voters:
I20260812 02:29:11.695004 13615 raft_consensus.cc:2781] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:11.694983 13363 raft_consensus.cc:2454] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fadcf3359c444a3394de3d8acad8ca89 in term 1.
I20260812 02:29:11.696235 13615 raft_consensus.cc:686] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [term 1 LEADER]: Becoming Leader. State: Replica: fadcf3359c444a3394de3d8acad8ca89, State: Running, Role: LEADER
I20260812 02:29:11.696985 13615 consensus_queue.cc:237] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } }
I20260812 02:29:11.705746 13020 catalog_manager.cc:5495] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89 reported cstate change: term changed from 0 to 1, leader changed from <none> to fadcf3359c444a3394de3d8acad8ca89 (127.12.8.193). New cstate: current_term: 1 leader_uuid: "fadcf3359c444a3394de3d8acad8ca89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadcf3359c444a3394de3d8acad8ca89" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad25e125bbad4326946ac055a19946bc" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04caee0064914ecca0f8ddb4a42ea1eb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 36323 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:29:11 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:11.758694 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.728847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.12.8.193:33995)
0812 02:29:11.729339 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.729348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.730610 (+ 1262us) server_negotiation.cc:407] Connection header received
0812 02:29:11.730876 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.730883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.730984 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.731172 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:11.734449 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.735650 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.737765 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.738646 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.738776 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.748964 (+ 10188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:11.748993 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:11.749016 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:11.749118 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:11.754122 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.755547 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.755560 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.755564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.756020 (+ 456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.756515 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.756519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.756522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.757075 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:11.757541 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.757869 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.758221 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
Aug 12 02:29:11 dist-test-slave-vq2q krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501751, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260812 02:29:11.808343 13257 tablet.cc:2369] T f4c3f7cc3faa4a149c965f120a0f1cfe P fadcf3359c444a3394de3d8acad8ca89: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 02:29:11 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501751, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:11.832304 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.812935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.12.8.193:33995)
0812 02:29:11.813254 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.813262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.813814 (+ 552us) server_negotiation.cc:407] Connection header received
0812 02:29:11.814062 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.814068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.814154 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.814246 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:11.815830 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.816972 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.818437 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.819333 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.819471 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.824117 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:11.824124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:11.824128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:11.824177 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:11.828565 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.829519 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.829524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.829527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.829829 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.830281 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.830285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.830287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.830691 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:11.831288 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.831433 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.831740 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 02:29:11.877369 13419 tablet.cc:2369] T f4c3f7cc3faa4a149c965f120a0f1cfe P 04caee0064914ecca0f8ddb4a42ea1eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 02:29:11 dist-test-slave-vq2q krb5kdc[12965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:29:11 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:11.913266 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.889971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.12.8.193:33995)
0812 02:29:11.890323 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.890331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.891079 (+ 748us) server_negotiation.cc:407] Connection header received
0812 02:29:11.891349 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.891354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.891516 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.891640 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:11.893533 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.894782 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.896725 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.897590 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.897725 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.904190 (+ 6465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:11.904197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:11.904201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:11.904253 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:11.908705 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.909985 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.909990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.909993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.910293 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.911007 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.911011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.911014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.911519 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:11.911926 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.912323 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.912646 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260812 02:29:11.916286 13151 tablet_service.cc:2908] Rejecting scan request for tablet f4c3f7cc3faa4a149c965f120a0f1cfe: Uninitialized: clean time has not yet been initialized
I20260812 02:29:11.921595 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 13096
I20260812 02:29:11.940204 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:11.923246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.12.8.193:33995)
0812 02:29:11.923509 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:11.923516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:11.923841 (+ 325us) server_negotiation.cc:407] Connection header received
0812 02:29:11.924022 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:11.924027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:11.924101 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:11.924196 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:11.926273 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.927036 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.928763 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:11.929619 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:11.929769 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:11.932832 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:11.932838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:11.932844 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:11.932886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:11.936520 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.937570 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.937575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.937577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.937752 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:11.938225 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:11.938230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:11.938234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:11.938648 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:11.938991 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:11.939261 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 02:29:11.939667 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 02:29:11.941469 13249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:12.041157 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 13096
I20260812 02:29:12.064482 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 13263
Aug 12 02:29:12 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:29:12.082298 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:12.065852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.12.8.194:36323)
0812 02:29:12.066067 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:12.066072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:12.066095 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:29:12.066168 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:12.066172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:12.066284 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:12.066386 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:12.067754 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:12.068637 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:12.070220 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:12.070761 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:12.070922 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:12.075941 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:12.075964 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:12.075983 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:12.076037 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:12.079244 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:12.080097 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:12.080105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:12.080107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:12.080251 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:12.080630 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:12.080633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:12.080635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:12.081065 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:12.081402 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:12.081720 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 02:29:12.081961 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:29:12.083407 13413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:12.184659 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 13263
I20260812 02:29:12.208457 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 13425
Aug 12 02:29:12 dist-test-slave-vq2q krb5kdc[12965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:29:12.227028 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:12.209560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.12.8.195:39475)
0812 02:29:12.210326 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:12.210333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:12.210369 (+ 36us) server_negotiation.cc:407] Connection header received
0812 02:29:12.210484 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:12.210491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:12.210601 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:12.210739 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:12.212200 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:12.213070 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:12.214648 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:12.215139 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:12.215268 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:12.220108 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:12.220132 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:12.220156 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:12.220218 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:12.223653 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:12.224603 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:12.224615 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:12.224617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:12.224794 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:12.225227 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:12.225234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:12.225238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:12.225681 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:12.226024 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:12.226338 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 02:29:12.226614 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":354,"threads_started":1}
I20260812 02:29:12.227893 13575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:12.321532 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 13425
I20260812 02:29:12.346175 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 12981
I20260812 02:29:12.347572 13059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:12.464146 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 12981
2026-08-12T02:29:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4857 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:12 dist-test-slave-vq2q krb5kdc[13645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:12 dist-test-slave-vq2q krb5kdc[13645](info): set up 2 sockets
Aug 12 02:29:12 dist-test-slave-vq2q krb5kdc[13645](info): commencing operation
krb5kdc: starting...
W20260812 02:29:13.599471 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:13 dist-test-slave-vq2q krb5kdc[13645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:13.862459 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:42225
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:42225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:14.288736 13659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:14.289160 13659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:14.289352 13659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:14.297676 13659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:14.297840 13659 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:29:14.297897 13659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:14.297993 13659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:14.298079 13659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:14.298172 13659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:14.309801 13659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:42225
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:42225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:14.311663 13659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:14.313387 13659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:14.327441 13665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:14.327602 13668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:14.327610 13664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:14.328686 13659 server_base.cc:1034] running on GCE node
I20260812 02:29:14.329748 13659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:14.331564 13659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:14.332818 13659 hybrid_clock.cc:648] HybridClock initialized: now 1786501754332730 us; error 57 us; skew 500 ppm
I20260812 02:29:14.333217 13659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:14 dist-test-slave-vq2q krb5kdc[13645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501754, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:14.336716 13659 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:14.338564 13659 webserver.cc:460] Webserver started at http://127.12.8.254:41525/ using document root <none> and password file <none>
I20260812 02:29:14.339435 13659 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:14.339542 13659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:14.339898 13659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:14.343219 13659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "355d9148a8ee411aaed4d0bb25e5de81"
format_stamp: "Formatted at 2026-08-12 02:29:14 on dist-test-slave-vq2q"
server_key: "d93341f2235536229f44b5d0f5104dd8"
server_key_iv: "a05548cfb2e2cf6a3ba0443da1c7e838"
server_key_version: "encryptionkey@0"
I20260812 02:29:14.344045 13659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "355d9148a8ee411aaed4d0bb25e5de81"
format_stamp: "Formatted at 2026-08-12 02:29:14 on dist-test-slave-vq2q"
server_key: "d93341f2235536229f44b5d0f5104dd8"
server_key_iv: "a05548cfb2e2cf6a3ba0443da1c7e838"
server_key_version: "encryptionkey@0"
I20260812 02:29:14.350219 13659 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 02:29:14.354420 13676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:14.356194 13659 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:14.356451 13659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "355d9148a8ee411aaed4d0bb25e5de81"
format_stamp: "Formatted at 2026-08-12 02:29:14 on dist-test-slave-vq2q"
server_key: "d93341f2235536229f44b5d0f5104dd8"
server_key_iv: "a05548cfb2e2cf6a3ba0443da1c7e838"
server_key_version: "encryptionkey@0"
I20260812 02:29:14.356706 13659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:14.381024 13659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:14.385803 13659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:14.386081 13659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:14.413348 13659 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:42225
I20260812 02:29:14.413447 13751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:42225 every 8 connection(s)
I20260812 02:29:14.414927 13659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:14.420051 13752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:14.421620 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 13659
I20260812 02:29:14.421865 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:14.422591 12323 external_mini_cluster.cc:1428] Setting key f3196bd8097f1c08b56e9ffadf3a67f2
I20260812 02:29:14.438221 13752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81: Bootstrap starting.
Aug 12 02:29:14 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:14.442862 13752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:14.444295 13752 log.cc:824] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:14.447889 13752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81: No bootstrap required, opened a new log
I20260812 02:29:14.449651 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:14.425226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.12.8.254:42225)
0812 02:29:14.426341 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:14.426361 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:14.426437 (+ 76us) server_negotiation.cc:407] Connection header received
0812 02:29:14.428551 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:14.428593 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:14.429392 (+ 799us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:14.430000 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:14.431412 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:14.433559 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:14.434974 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:14.436304 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:14.436551 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:14.440816 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:14.440830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:14.440854 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:14.440927 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:14.444991 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:14.445794 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:14.445802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:14.445810 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:14.446018 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:14.446389 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:14.446393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:14.446395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:14.447043 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:14.447369 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:14.447835 (+ 466us) server_negotiation.cc:299] Negotiation successful
0812 02:29:14.448363 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":455,"threads_started":1}
I20260812 02:29:14.456822 13752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355d9148a8ee411aaed4d0bb25e5de81" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42225 } }
I20260812 02:29:14.457396 13752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:14.457499 13752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355d9148a8ee411aaed4d0bb25e5de81, State: Initialized, Role: FOLLOWER
I20260812 02:29:14.458226 13752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355d9148a8ee411aaed4d0bb25e5de81" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42225 } }
I20260812 02:29:14.458516 13752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:14.458685 13752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:14.458909 13752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:14.461812 13752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355d9148a8ee411aaed4d0bb25e5de81" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42225 } }
I20260812 02:29:14.462495 13752 leader_election.cc:302] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 355d9148a8ee411aaed4d0bb25e5de81; no voters:
I20260812 02:29:14.463220 13752 leader_election.cc:288] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:14.463444 13760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:14.464648 13760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [term 1 LEADER]: Becoming Leader. State: Replica: 355d9148a8ee411aaed4d0bb25e5de81, State: Running, Role: LEADER
I20260812 02:29:14.465492 13760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355d9148a8ee411aaed4d0bb25e5de81" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42225 } }
I20260812 02:29:14.465994 13752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:14.469094 13761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "355d9148a8ee411aaed4d0bb25e5de81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355d9148a8ee411aaed4d0bb25e5de81" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42225 } } }
I20260812 02:29:14.469512 13762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355d9148a8ee411aaed4d0bb25e5de81. Latest consensus state: current_term: 1 leader_uuid: "355d9148a8ee411aaed4d0bb25e5de81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "355d9148a8ee411aaed4d0bb25e5de81" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42225 } } }
I20260812 02:29:14.470549 13761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:14.470572 13762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:14.472208 13768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:14.477816 13768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:14.487429 13768 catalog_manager.cc:1347] Generated new cluster ID: 3c9ead600e76471a8f3ddc36f0b24a02
I20260812 02:29:14.487589 13768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:14.508714 13768 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:14.510303 13768 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:14.522159 13768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81: Generated new TSK 0
I20260812 02:29:14.523664 13768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:14.581729 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42225
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:14.959791 13784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:14.960103 13784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:14.960233 13784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:14.967975 13784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:14.968106 13784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:14.968223 13784 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:29:14.968329 13784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:14.978927 13784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:42225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:14.980382 13784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:14.981933 13784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:14.995289 13792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:14.995198 13794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:14.995321 13791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:14.996148 13784 server_base.cc:1034] running on GCE node
I20260812 02:29:14.996704 13784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:14.997629 13784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:14.998868 13784 hybrid_clock.cc:648] HybridClock initialized: now 1786501754998799 us; error 32 us; skew 500 ppm
I20260812 02:29:14.999166 13784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:14 dist-test-slave-vq2q krb5kdc[13645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501754, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:15.002517 13784 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:15.004071 13784 webserver.cc:460] Webserver started at http://127.12.8.193:33775/ using document root <none> and password file <none>
I20260812 02:29:15.004823 13784 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:15.004931 13784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:15.005306 13784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:15.008496 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "24ba67f5b891422faa5ec24c09e626dc"
format_stamp: "Formatted at 2026-08-12 02:29:15 on dist-test-slave-vq2q"
server_key: "954d14313ab7aa782465a0f16f2de82e"
server_key_iv: "ebb9822793c65a0d44d42a9ef56fac52"
server_key_version: "encryptionkey@0"
I20260812 02:29:15.009500 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "24ba67f5b891422faa5ec24c09e626dc"
format_stamp: "Formatted at 2026-08-12 02:29:15 on dist-test-slave-vq2q"
server_key: "954d14313ab7aa782465a0f16f2de82e"
server_key_iv: "ebb9822793c65a0d44d42a9ef56fac52"
server_key_version: "encryptionkey@0"
I20260812 02:29:15.015241 13784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:29:15.019049 13802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:15.020327 13784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:29:15.020488 13784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "24ba67f5b891422faa5ec24c09e626dc"
format_stamp: "Formatted at 2026-08-12 02:29:15 on dist-test-slave-vq2q"
server_key: "954d14313ab7aa782465a0f16f2de82e"
server_key_iv: "ebb9822793c65a0d44d42a9ef56fac52"
server_key_version: "encryptionkey@0"
I20260812 02:29:15.020718 13784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:15.032702 13784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:15.036891 13784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:15.037160 13784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:15.040126 13784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:15.040213 13784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:15.040352 13784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:15.040412 13784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:29:15 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501754, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:15.066480 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:15.041005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:58415 (local address 127.12.8.254:42225)
0812 02:29:15.041521 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:15.041527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:15.042766 (+ 1239us) server_negotiation.cc:407] Connection header received
0812 02:29:15.045698 (+ 2932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:15.045718 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:15.045840 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:15.045929 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:15.049275 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.050247 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.052124 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.052571 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.052700 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:15.058371 (+ 5671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:15.058380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:15.058385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:15.058436 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:15.061823 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:15.063066 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:15.063072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:15.063075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:15.063244 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:15.063933 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:15.063938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:15.063942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:15.064362 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:15.064544 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:15.065789 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0812 02:29:15.066081 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":270,"threads_started":1}
I20260812 02:29:15.068420 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:15.041651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42225 (local address 127.12.8.193:58415)
0812 02:29:15.042566 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:15.042616 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:15.045315 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:15.046637 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:15.046652 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:15.047889 (+ 1237us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:15.049023 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:15.049057 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.050411 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.050420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:15.051977 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:15.051990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.052870 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.052879 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:15.053221 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:15.054151 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:15.054187 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:15.058018 (+ 3831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:15.062014 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:15.062031 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:15.062053 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:15.062901 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:15.063376 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:15.063382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:15.063389 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:15.063734 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:15.064524 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:15.064534 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:15.065217 (+ 683us) client_negotiation.cc:769] Sending connection context
0812 02:29:15.065633 (+ 416us) client_negotiation.cc:240] Negotiation successful
0812 02:29:15.066336 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":381,"threads_started":1}
I20260812 02:29:15.080938 13784 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:44411
I20260812 02:29:15.080960 13952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:44411 every 8 connection(s)
I20260812 02:29:15.082439 13784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:15.085000 13953 heartbeater.cc:347] Connected to a master server at 127.12.8.254:42225
I20260812 02:29:15.085243 13953 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:15.085907 13953 heartbeater.cc:511] Master 127.12.8.254:42225 requested a full tablet report, sending...
I20260812 02:29:15.086712 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 13784
I20260812 02:29:15.086939 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:15.087709 12323 external_mini_cluster.cc:1428] Setting key bf673e1b109d80520e4f8adb4507c204
I20260812 02:29:15.088455 13700 ts_manager.cc:194] Registered new tserver with Master: 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193:44411)
I20260812 02:29:15.090533 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:15.076346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42225 (local address 127.12.8.193:42105)
0812 02:29:15.076972 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:15.076987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:15.077131 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:15.077685 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:15.077690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:15.078345 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:15.078848 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:15.078867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.079911 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.079916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:15.081641 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:15.081657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.082483 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.082491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:15.082669 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:15.083347 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:15.083378 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:15.084537 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:15.087765 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:15.087770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:15.087774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:15.088166 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:15.088617 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:15.088622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:15.088625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:15.088794 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:15.089388 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:15.089393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:15.089589 (+ 196us) client_negotiation.cc:769] Sending connection context
0812 02:29:15.089713 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 02:29:15.090056 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":362,"threads_started":1}
I20260812 02:29:15.090538 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:15.076220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:42105 (local address 127.12.8.254:42225)
0812 02:29:15.076393 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:15.076398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:15.077070 (+ 672us) server_negotiation.cc:407] Connection header received
0812 02:29:15.077296 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:15.077302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:15.077387 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:15.077487 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:15.078997 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.079754 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.081809 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.082273 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.082375 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:15.084701 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:15.084706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:15.084709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:15.084742 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:15.087624 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:15.088286 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:15.088292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:15.088295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:15.088428 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:15.088901 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:15.088906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:15.088910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:15.089275 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:15.089463 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:15.089969 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 02:29:15.090196 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:29:15.090493 13700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:58415
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:15.136843 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42225
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:15.493162 13957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:15.493497 13957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:15.493613 13957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:15.500943 13957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:15.501084 13957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:15.501191 13957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:29:15.501305 13957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:15.511438 13957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:42225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:15.512763 13957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:15.514081 13957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:15.527187 13962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:15.527107 13963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:15.527153 13965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:15.528035 13957 server_base.cc:1034] running on GCE node
I20260812 02:29:15.528731 13957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:15.529855 13957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:15.531087 13957 hybrid_clock.cc:648] HybridClock initialized: now 1786501755531001 us; error 52 us; skew 500 ppm
I20260812 02:29:15.531405 13957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:15 dist-test-slave-vq2q krb5kdc[13645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501755, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:15.534605 13957 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:15.536178 13957 webserver.cc:460] Webserver started at http://127.12.8.194:34615/ using document root <none> and password file <none>
I20260812 02:29:15.536902 13957 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:15.537006 13957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:15.537385 13957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:15.540359 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "4669dd0b780f4514a6a96413dd5a4234"
format_stamp: "Formatted at 2026-08-12 02:29:15 on dist-test-slave-vq2q"
server_key: "81a68f3b83b1451de7973e86763d5ef6"
server_key_iv: "ea4242b54e3ec26f9b0876ddc89e853f"
server_key_version: "encryptionkey@0"
I20260812 02:29:15.541204 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "4669dd0b780f4514a6a96413dd5a4234"
format_stamp: "Formatted at 2026-08-12 02:29:15 on dist-test-slave-vq2q"
server_key: "81a68f3b83b1451de7973e86763d5ef6"
server_key_iv: "ea4242b54e3ec26f9b0876ddc89e853f"
server_key_version: "encryptionkey@0"
I20260812 02:29:15.546482 13957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 02:29:15.550071 13974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:15.551471 13957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:29:15.551637 13957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "4669dd0b780f4514a6a96413dd5a4234"
format_stamp: "Formatted at 2026-08-12 02:29:15 on dist-test-slave-vq2q"
server_key: "81a68f3b83b1451de7973e86763d5ef6"
server_key_iv: "ea4242b54e3ec26f9b0876ddc89e853f"
server_key_version: "encryptionkey@0"
I20260812 02:29:15.551860 13957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:15.571913 13957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:15.576313 13957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:15.576572 13957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:15.579433 13957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:15.579535 13957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:15.579684 13957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:15.579803 13957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:29:15 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501755, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:15.601046 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:15.580062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:44523 (local address 127.12.8.254:42225)
0812 02:29:15.580285 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:15.580291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:15.581773 (+ 1482us) server_negotiation.cc:407] Connection header received
0812 02:29:15.583928 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:15.583935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:15.584024 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:15.584111 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:15.586290 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.587039 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.588628 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.589068 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.589223 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:15.594330 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:15.594337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:15.594341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:15.594383 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:15.597453 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:15.598454 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:15.598460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:15.598464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:15.598641 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:15.599079 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:15.599084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:15.599087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:15.599498 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:15.599670 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:15.600354 (+ 684us) server_negotiation.cc:299] Negotiation successful
0812 02:29:15.600619 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:29:15.601891 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:15.580690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42225 (local address 127.12.8.194:44523)
0812 02:29:15.581604 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:15.581653 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:15.583657 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:15.584406 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:15.584418 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:15.585160 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:15.586117 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:15.586137 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.587188 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.587193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:15.588486 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:15.588499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.589399 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.589407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:15.589705 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:15.590563 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:15.590603 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:15.594060 (+ 3457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:15.597610 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:15.597618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:15.597633 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:15.598303 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:15.598766 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:15.598770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:15.598772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:15.598977 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:15.599623 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:15.599628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:15.599985 (+ 357us) client_negotiation.cc:769] Sending connection context
0812 02:29:15.600230 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 02:29:15.600654 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":426,"threads_started":1}
I20260812 02:29:15.621908 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:15.607312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:41219 (local address 127.12.8.254:42225)
0812 02:29:15.607529 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:15.607536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:15.607998 (+ 462us) server_negotiation.cc:407] Connection header received
0812 02:29:15.608156 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:15.608159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:15.608218 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:15.608304 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:15.609883 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.610803 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.612775 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.613283 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.613389 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:15.616041 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:15.616048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:15.616053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:15.616101 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:15.619020 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:15.619842 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:15.619846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:15.619848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:15.619976 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:15.620316 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:15.620319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:15.620320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:15.620645 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:15.620758 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:15.621308 (+ 550us) server_negotiation.cc:299] Negotiation successful
0812 02:29:15.621570 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:29:15.622139 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:15.607208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42225 (local address 127.12.8.194:41219)
0812 02:29:15.607886 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:15.607913 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:15.608058 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:15.608363 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:15.608368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:15.609087 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:15.609720 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:15.609754 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.610953 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.610959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:15.612624 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:15.612641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:15.613522 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:15.613530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:15.613777 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:15.614446 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:15.614480 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:15.615843 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:15.619148 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:15.619154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:15.619158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:15.619737 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:15.620065 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:15.620070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:15.620074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:15.620225 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:15.620738 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:15.620744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:15.620958 (+ 214us) client_negotiation.cc:769] Sending connection context
0812 02:29:15.621131 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 02:29:15.621589 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":351,"threads_started":1}
I20260812 02:29:15.636269 13957 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:42035
I20260812 02:29:15.636292 14118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:42035 every 8 connection(s)
I20260812 02:29:15.637530 13957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:15.639798 14121 heartbeater.cc:347] Connected to a master server at 127.12.8.254:42225
I20260812 02:29:15.640067 14121 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:15.640311 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 13957
I20260812 02:29:15.640491 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:15.640844 14121 heartbeater.cc:511] Master 127.12.8.254:42225 requested a full tablet report, sending...
I20260812 02:29:15.640941 12323 external_mini_cluster.cc:1428] Setting key ab8ca511a99b6f37cdbd14ac5c1774dc
I20260812 02:29:15.642344 13698 ts_manager.cc:194] Registered new tserver with Master: 4669dd0b780f4514a6a96413dd5a4234 (127.12.8.194:42035)
I20260812 02:29:15.643323 13698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:44523
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:15.688653 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42225
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:29:16.094666 13953 heartbeater.cc:503] Master 127.12.8.254:42225 was elected leader, sending a full tablet report...
W20260812 02:29:16.134277 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:16.134698 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:16.134819 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:16.143168 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:16.143348 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:16.143455 14126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:29:16.143566 14126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:16.155098 14126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:42225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:16.157027 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:16.158867 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:16.176052 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:16.176288 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:16.176355 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:16.179185 14126 server_base.cc:1034] running on GCE node
I20260812 02:29:16.180032 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:16.181321 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:16.182603 14126 hybrid_clock.cc:648] HybridClock initialized: now 1786501756182515 us; error 52 us; skew 500 ppm
I20260812 02:29:16.182986 14126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:16 dist-test-slave-vq2q krb5kdc[13645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501756, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:16.187163 14126 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:29:16.189283 14126 webserver.cc:460] Webserver started at http://127.12.8.195:33355/ using document root <none> and password file <none>
I20260812 02:29:16.190256 14126 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:16.190377 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:16.190768 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:16.194888 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "2a79f429239646f1a07249872349f7f3"
format_stamp: "Formatted at 2026-08-12 02:29:16 on dist-test-slave-vq2q"
server_key: "0acd7dc746464c636e438a2538f41798"
server_key_iv: "867d4d26a6867a02a52d5001580f0ad8"
server_key_version: "encryptionkey@0"
I20260812 02:29:16.196059 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "2a79f429239646f1a07249872349f7f3"
format_stamp: "Formatted at 2026-08-12 02:29:16 on dist-test-slave-vq2q"
server_key: "0acd7dc746464c636e438a2538f41798"
server_key_iv: "867d4d26a6867a02a52d5001580f0ad8"
server_key_version: "encryptionkey@0"
I20260812 02:29:16.203470 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260812 02:29:16.208257 14146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:16.210258 14126 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:29:16.210502 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "2a79f429239646f1a07249872349f7f3"
format_stamp: "Formatted at 2026-08-12 02:29:16 on dist-test-slave-vq2q"
server_key: "0acd7dc746464c636e438a2538f41798"
server_key_iv: "867d4d26a6867a02a52d5001580f0ad8"
server_key_version: "encryptionkey@0"
I20260812 02:29:16.210784 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:16.228196 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:16.233551 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:16.233865 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:16.237346 14126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:16.237475 14126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:16.237663 14126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:16.237738 14126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:29:16 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501756, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:16.263463 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.238147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:41835 (local address 127.12.8.254:42225)
0812 02:29:16.238860 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:16.238865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:16.240494 (+ 1629us) server_negotiation.cc:407] Connection header received
0812 02:29:16.243104 (+ 2610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:16.243110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:16.243224 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:16.243337 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:16.246086 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.247101 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.250149 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.250585 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.250700 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.256508 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:16.256514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:16.256518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:16.256559 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:16.259410 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:16.260235 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:16.260240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:16.260242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:16.260384 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:16.260852 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:16.260855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:16.260857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:16.261287 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:16.261447 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:16.262962 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0812 02:29:16.263146 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":455,"threads_started":1}
I20260812 02:29:16.266268 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.239092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42225 (local address 127.12.8.195:41835)
0812 02:29:16.240215 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:16.240270 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:16.242758 (+ 2488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:16.243696 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:16.243724 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:16.244693 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:16.245858 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.245893 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.247445 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.247457 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.249963 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.249990 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.250881 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.250890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.251175 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.252181 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:16.252233 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:16.256215 (+ 3982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:16.259559 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:16.259570 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:16.259588 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:16.260106 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:16.260491 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:16.260497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:16.260501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:16.260725 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:16.261522 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:16.261537 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:16.262360 (+ 823us) client_negotiation.cc:769] Sending connection context
0812 02:29:16.262847 (+ 487us) client_negotiation.cc:240] Negotiation successful
0812 02:29:16.263721 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":463,"threads_started":1}
I20260812 02:29:16.300896 14126 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:33117
I20260812 02:29:16.300913 14299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:33117 every 8 connection(s)
I20260812 02:29:16.302800 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:16.303755 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.287536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:50175 (local address 127.12.8.254:42225)
0812 02:29:16.287736 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:16.287742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:16.288934 (+ 1192us) server_negotiation.cc:407] Connection header received
0812 02:29:16.289290 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:16.289297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:16.289401 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:16.289534 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:16.291340 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.292083 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.293510 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.294050 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.294167 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.297635 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:16.297642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:16.297647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:16.297684 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:16.300707 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:16.301598 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:16.301603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:16.301607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:16.301742 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:16.302113 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:16.302116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:16.302119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:16.302450 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:16.302597 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:16.303171 (+ 574us) server_negotiation.cc:299] Negotiation successful
0812 02:29:16.303401 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:29:16.304075 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.287934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42225 (local address 127.12.8.195:50175)
0812 02:29:16.288753 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:16.288783 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:16.289035 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:16.289797 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:16.289801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:16.290595 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:16.291166 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.291179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.292237 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.292241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.293370 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.293383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.294273 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.294284 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.294425 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.295413 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:16.295457 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:16.297312 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:16.300862 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:16.300869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:16.300874 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:16.301475 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:16.301862 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:16.301867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:16.301871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:16.301997 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:16.302570 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:16.302575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:16.302759 (+ 184us) client_negotiation.cc:769] Sending connection context
0812 02:29:16.302898 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 02:29:16.303308 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":406,"threads_started":1}
I20260812 02:29:16.306569 14300 heartbeater.cc:347] Connected to a master server at 127.12.8.254:42225
I20260812 02:29:16.306883 14300 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:16.307914 14300 heartbeater.cc:511] Master 127.12.8.254:42225 requested a full tablet report, sending...
I20260812 02:29:16.309067 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 14126
I20260812 02:29:16.309331 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:16.309783 12323 external_mini_cluster.cc:1428] Setting key 20e757ed6c6c66494469a00f12de3db2
I20260812 02:29:16.309923 13698 ts_manager.cc:194] Registered new tserver with Master: 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:16.311028 13698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:41835
I20260812 02:29:16.313746 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:29:16 dist-test-slave-vq2q krb5kdc[13645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:29:16 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:16.378515 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.360908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.12.8.254:42225)
0812 02:29:16.361170 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:16.361207 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:16.361652 (+ 445us) server_negotiation.cc:407] Connection header received
0812 02:29:16.361898 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:16.361903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:16.362002 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:16.362105 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:16.363816 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.364628 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.366180 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.366709 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.366821 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.372057 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:16.372065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:16.372069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:16.372125 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:16.375283 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:16.376177 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:16.376183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:16.376185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:16.376345 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:16.376801 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:16.376805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:16.376807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:16.377221 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:16.377573 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:16.377854 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 02:29:16.378119 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 02:29:16.384050 13698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:16.392221 13698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:16.425843 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.409352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:44411 (local address 127.0.0.1:41972)
0812 02:29:16.410401 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:16.410463 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:16.410784 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:16.412567 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:16.412578 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:16.412624 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:16.413279 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.413307 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.417600 (+ 4293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.417610 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.421215 (+ 3605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.421248 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.424224 (+ 2976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.424234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.424615 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.424710 (+ 95us) client_negotiation.cc:769] Sending connection context
0812 02:29:16.424890 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 02:29:16.425232 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":466,"threads_started":1}
I20260812 02:29:16.428016 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.409616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.12.8.193:44411)
0812 02:29:16.411600 (+ 1984us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:16.411613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:16.411655 (+ 42us) server_negotiation.cc:407] Connection header received
0812 02:29:16.411804 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:16.411815 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:16.412187 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:16.413601 (+ 1414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:16.413932 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.417341 (+ 3409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.421689 (+ 4348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.423858 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.426878 (+ 3020us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.427232 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:16.427343 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 02:29:16.427615 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":2182,"threads_started":1}
I20260812 02:29:16.433028 13863 tablet_service.cc:1467] Processing CreateTablet for tablet 954ecc61d5ae41eabf302d79de2ff0d8 (DEFAULT_TABLE table=test-table [id=feb4728c3ccd4a66ba08ae9d3b361710]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:16.434787 13863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954ecc61d5ae41eabf302d79de2ff0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:16.443274 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.410405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:33117 (local address 127.0.0.1:51160)
0812 02:29:16.426430 (+ 16025us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:16.426462 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:16.426646 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:16.435126 (+ 8480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:16.435133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:16.435176 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:16.435424 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.435438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.438152 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.438158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.440339 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.440358 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.442411 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.442417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.442603 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.442649 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 02:29:16.442725 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:29:16.442903 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15825,"thread_start_us":18757,"threads_started":1}
I20260812 02:29:16.443686 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.409444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42035 (local address 127.0.0.1:50210)
0812 02:29:16.415413 (+ 5969us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:16.415446 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:16.415813 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:16.435870 (+ 20057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:16.435876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:16.435905 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:16.436399 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.436414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.438494 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.438500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.440868 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.440882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.442724 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.442730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.442904 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.442932 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:29:16.442988 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 02:29:16.443119 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5734,"thread_start_us":5553,"threads_started":1}
I20260812 02:29:16.444213 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.410322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.12.8.194:42035)
0812 02:29:16.435024 (+ 24702us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:16.435033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:16.435064 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:29:16.435184 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:16.435191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:16.435430 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:16.436660 (+ 1230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:16.436925 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.438346 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.441055 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.442463 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.443368 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.443633 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:16.443725 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 02:29:16.443919 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24529,"thread_start_us":24388,"threads_started":1}
I20260812 02:29:16.445636 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.411455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.12.8.195:33117)
0812 02:29:16.433209 (+ 21754us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:16.433223 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:16.433281 (+ 58us) server_negotiation.cc:407] Connection header received
0812 02:29:16.433442 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:16.433454 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:16.433861 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:16.435650 (+ 1789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:16.436056 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.437954 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.440583 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.442197 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.444097 (+ 1900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.444984 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:16.445074 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 02:29:16.445293 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21440,"thread_start_us":22777,"threads_started":1}
I20260812 02:29:16.448593 14047 tablet_service.cc:1467] Processing CreateTablet for tablet 954ecc61d5ae41eabf302d79de2ff0d8 (DEFAULT_TABLE table=test-table [id=feb4728c3ccd4a66ba08ae9d3b361710]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:16.449432 14217 tablet_service.cc:1467] Processing CreateTablet for tablet 954ecc61d5ae41eabf302d79de2ff0d8 (DEFAULT_TABLE table=test-table [id=feb4728c3ccd4a66ba08ae9d3b361710]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:16.450253 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954ecc61d5ae41eabf302d79de2ff0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:16.451057 14217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954ecc61d5ae41eabf302d79de2ff0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:16.451406 14325 tablet_bootstrap.cc:492] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Bootstrap starting.
I20260812 02:29:16.455936 14325 tablet_bootstrap.cc:654] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:16.457674 14325 log.cc:824] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:16.463380 14327 tablet_bootstrap.cc:492] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Bootstrap starting.
I20260812 02:29:16.468662 14325 tablet_bootstrap.cc:492] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: No bootstrap required, opened a new log
I20260812 02:29:16.468809 14327 tablet_bootstrap.cc:654] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:16.469332 14325 ts_tablet_manager.cc:1397] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20260812 02:29:16.469715 14328 tablet_bootstrap.cc:492] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Bootstrap starting.
I20260812 02:29:16.471416 14327 log.cc:824] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:16.476413 14327 tablet_bootstrap.cc:492] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: No bootstrap required, opened a new log
I20260812 02:29:16.476893 14327 ts_tablet_manager.cc:1397] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260812 02:29:16.476926 14328 tablet_bootstrap.cc:654] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:16.479048 14328 log.cc:824] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:16.484092 14325 raft_consensus.cc:348] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.484768 14328 tablet_bootstrap.cc:492] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: No bootstrap required, opened a new log
I20260812 02:29:16.484805 14325 raft_consensus.cc:374] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:16.484978 14325 raft_consensus.cc:729] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ba67f5b891422faa5ec24c09e626dc, State: Initialized, Role: FOLLOWER
I20260812 02:29:16.485263 14328 ts_tablet_manager.cc:1397] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260812 02:29:16.486320 14325 consensus_queue.cc:260] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.492616 14327 raft_consensus.cc:348] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.493353 14327 raft_consensus.cc:374] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:16.493520 14327 raft_consensus.cc:729] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669dd0b780f4514a6a96413dd5a4234, State: Initialized, Role: FOLLOWER
I20260812 02:29:16.494594 14327 consensus_queue.cc:260] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.496408 14325 ts_tablet_manager.cc:1428] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Time spent starting tablet: real 0.027s user 0.014s sys 0.009s
I20260812 02:29:16.503435 14121 heartbeater.cc:503] Master 127.12.8.254:42225 was elected leader, sending a full tablet report...
I20260812 02:29:16.503347 14328 raft_consensus.cc:348] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.504333 14328 raft_consensus.cc:374] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:16.504575 14328 raft_consensus.cc:729] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a79f429239646f1a07249872349f7f3, State: Initialized, Role: FOLLOWER
I20260812 02:29:16.505154 14327 ts_tablet_manager.cc:1428] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Time spent starting tablet: real 0.028s user 0.012s sys 0.016s
I20260812 02:29:16.505875 14328 consensus_queue.cc:260] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.508366 14300 heartbeater.cc:503] Master 127.12.8.254:42225 was elected leader, sending a full tablet report...
I20260812 02:29:16.509196 14328 ts_tablet_manager.cc:1428] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Time spent starting tablet: real 0.024s user 0.010s sys 0.012s
W20260812 02:29:16.555614 14301 tablet.cc:2369] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:29:16.585878 13954 tablet.cc:2369] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:29:16.640275 14122 tablet.cc:2369] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:16.844682 14334 raft_consensus.cc:482] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:16.845170 14334 raft_consensus.cc:504] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.848129 14334 leader_election.cc:288] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193:44411), 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:16.862624 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.848555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:35799 (local address 127.12.8.193:44411)
0812 02:29:16.848809 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:16.848814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:16.852083 (+ 3269us) server_negotiation.cc:407] Connection header received
0812 02:29:16.852335 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:16.852342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:16.852504 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:16.852584 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:16.853943 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.855291 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.859357 (+ 4066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.861338 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.861573 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.861671 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:16.862236 (+ 565us) server_negotiation.cc:299] Negotiation successful
0812 02:29:16.862373 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:29:16.862954 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.848966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:44411 (local address 127.12.8.194:35799)
0812 02:29:16.851899 (+ 2933us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:16.851919 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:16.852122 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:16.852631 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:16.852637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:16.852668 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:16.853732 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.853751 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.855479 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.855485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.859085 (+ 3600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.859103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.861753 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.861761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.862011 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.862052 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 02:29:16.862161 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 02:29:16.862322 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2670,"thread_start_us":1940,"threads_started":1}
I20260812 02:29:16.864579 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.848892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:50389 (local address 127.12.8.195:33117)
0812 02:29:16.849199 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:16.849208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:16.852232 (+ 3024us) server_negotiation.cc:407] Connection header received
0812 02:29:16.852455 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:16.852459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:16.852539 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:16.852658 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:16.854255 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.855331 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.859469 (+ 4138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.861338 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.861588 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.861671 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:16.863904 (+ 2233us) server_negotiation.cc:299] Negotiation successful
0812 02:29:16.864097 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 02:29:16.864609 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:16.849672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:33117 (local address 127.12.8.194:50389)
0812 02:29:16.852018 (+ 2346us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:16.852046 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:16.852294 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:16.852702 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:16.852707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:16.852735 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:16.853914 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.853931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.855495 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.855499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.859144 (+ 3645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:16.859164 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:16.863455 (+ 4291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:16.863463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:16.863663 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:16.863695 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:29:16.863788 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 02:29:16.863958 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2139,"thread_start_us":1589,"threads_started":1}
I20260812 02:29:16.864837 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954ecc61d5ae41eabf302d79de2ff0d8" candidate_uuid: "4669dd0b780f4514a6a96413dd5a4234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ba67f5b891422faa5ec24c09e626dc" is_pre_election: true
I20260812 02:29:16.865913 13887 raft_consensus.cc:2454] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4669dd0b780f4514a6a96413dd5a4234 in term 0.
I20260812 02:29:16.866328 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954ecc61d5ae41eabf302d79de2ff0d8" candidate_uuid: "4669dd0b780f4514a6a96413dd5a4234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a79f429239646f1a07249872349f7f3" is_pre_election: true
I20260812 02:29:16.867434 13977 leader_election.cc:302] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ba67f5b891422faa5ec24c09e626dc, 4669dd0b780f4514a6a96413dd5a4234; no voters:
I20260812 02:29:16.867745 14243 raft_consensus.cc:2454] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4669dd0b780f4514a6a96413dd5a4234 in term 0.
I20260812 02:29:16.868209 14334 raft_consensus.cc:2781] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:16.868516 14334 raft_consensus.cc:482] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:16.868712 14334 raft_consensus.cc:3037] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:16.961725 14334 raft_consensus.cc:504] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.962731 14334 leader_election.cc:288] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [CANDIDATE]: Term 1 election: Requested vote from peers 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193:44411), 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:16.963799 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954ecc61d5ae41eabf302d79de2ff0d8" candidate_uuid: "4669dd0b780f4514a6a96413dd5a4234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a79f429239646f1a07249872349f7f3"
I20260812 02:29:16.964140 14243 raft_consensus.cc:3037] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:16.964119 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954ecc61d5ae41eabf302d79de2ff0d8" candidate_uuid: "4669dd0b780f4514a6a96413dd5a4234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ba67f5b891422faa5ec24c09e626dc"
I20260812 02:29:16.964380 13887 raft_consensus.cc:3037] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:16.967780 14243 raft_consensus.cc:2454] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4669dd0b780f4514a6a96413dd5a4234 in term 1.
I20260812 02:29:16.968598 13978 leader_election.cc:302] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a79f429239646f1a07249872349f7f3, 4669dd0b780f4514a6a96413dd5a4234; no voters:
I20260812 02:29:16.969085 14334 raft_consensus.cc:2781] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:16.969758 14334 raft_consensus.cc:686] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 LEADER]: Becoming Leader. State: Replica: 4669dd0b780f4514a6a96413dd5a4234, State: Running, Role: LEADER
I20260812 02:29:16.970515 14334 consensus_queue.cc:237] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:16.976109 13700 catalog_manager.cc:5495] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4669dd0b780f4514a6a96413dd5a4234 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "4669dd0b780f4514a6a96413dd5a4234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } health_report { overall_health: UNKNOWN } } }
I20260812 02:29:17.061012 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.049872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.12.8.194:42035)
0812 02:29:17.050862 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:17.050871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:17.051413 (+ 542us) server_negotiation.cc:407] Connection header received
0812 02:29:17.051865 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:17.051870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:17.051962 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:17.052168 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:17.053006 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.054152 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.056963 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.057772 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.058031 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.059062 (+ 1031us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:17.059780 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:17.059918 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:17.060157 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 02:29:17.060240 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":553,"threads_started":1}
I20260812 02:29:17.065584 13887 raft_consensus.cc:2454] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4669dd0b780f4514a6a96413dd5a4234 in term 1.
I20260812 02:29:17.077621 14243 raft_consensus.cc:1261] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Refusing update from remote peer 4669dd0b780f4514a6a96413dd5a4234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:17.079610 13887 raft_consensus.cc:1261] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Refusing update from remote peer 4669dd0b780f4514a6a96413dd5a4234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:17.079401 14339 consensus_queue.cc:1035] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 02:29:17.081113 14334 consensus_queue.cc:1035] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:17.100881 14348 mvcc.cc:205] Tried to move back new op lower bound from 7317511196959264768 to 7317511196555755520. Current Snapshot: MvccSnapshot[applied={T|T < 7317511196959264768}]
I20260812 02:29:17.135948 13700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:17.136615 13700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:17.143149 13700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81: Sending DeleteTablet for 3 replicas of tablet 954ecc61d5ae41eabf302d79de2ff0d8
I20260812 02:29:17.145160 14047 tablet_service.cc:1514] Processing DeleteTablet for tablet 954ecc61d5ae41eabf302d79de2ff0d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:17 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:50210
I20260812 02:29:17.146373 13863 tablet_service.cc:1514] Processing DeleteTablet for tablet 954ecc61d5ae41eabf302d79de2ff0d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:17 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:41972
I20260812 02:29:17.148384 14217 tablet_service.cc:1514] Processing DeleteTablet for tablet 954ecc61d5ae41eabf302d79de2ff0d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:17 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51160
I20260812 02:29:17.148641 13700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:17.149849 13700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:17.161473 14367 tablet_replica.cc:330] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: stopping tablet replica
I20260812 02:29:17.162528 14367 raft_consensus.cc:2229] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:17.162617 14368 tablet_replica.cc:330] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: stopping tablet replica
I20260812 02:29:17.161634 13861 tablet_service.cc:1467] Processing CreateTablet for tablet 46e70ee08d3e46dd99da4b3a6c21e097 (DEFAULT_TABLE table=test-table [id=414a647dd984479295c5e47386886e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:17.163635 14367 raft_consensus.cc:2258] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:17.164057 13861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e70ee08d3e46dd99da4b3a6c21e097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:17.153139 14366 tablet_replica.cc:330] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: stopping tablet replica
I20260812 02:29:17.161324 14047 tablet_service.cc:1467] Processing CreateTablet for tablet 46e70ee08d3e46dd99da4b3a6c21e097 (DEFAULT_TABLE table=test-table [id=414a647dd984479295c5e47386886e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:17.165927 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e70ee08d3e46dd99da4b3a6c21e097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:17.166152 14366 raft_consensus.cc:2229] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:29:17.167981 14366 raft_consensus.cc:2258] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:17.170516 14367 ts_tablet_manager.cc:1905] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:17.171736 14366 ts_tablet_manager.cc:1905] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:17.176846 14215 tablet_service.cc:1467] Processing CreateTablet for tablet 46e70ee08d3e46dd99da4b3a6c21e097 (DEFAULT_TABLE table=test-table [id=414a647dd984479295c5e47386886e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:17.177832 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e70ee08d3e46dd99da4b3a6c21e097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:17.184432 14369 tablet_bootstrap.cc:492] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc: Bootstrap starting.
I20260812 02:29:17.188629 14369 tablet_bootstrap.cc:654] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:17.189565 14368 raft_consensus.cc:2229] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:17.190649 14368 raft_consensus.cc:2258] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:17.193774 14368 ts_tablet_manager.cc:1905] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:17.194495 14370 tablet_bootstrap.cc:492] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: Bootstrap starting.
I20260812 02:29:17.205749 14370 tablet_bootstrap.cc:654] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:17.208825 14373 tablet_bootstrap.cc:492] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: Bootstrap starting.
I20260812 02:29:17.210747 14369 tablet_bootstrap.cc:492] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc: No bootstrap required, opened a new log
I20260812 02:29:17.211052 14369 ts_tablet_manager.cc:1397] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.013s
I20260812 02:29:17.212256 14370 tablet_bootstrap.cc:492] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: No bootstrap required, opened a new log
I20260812 02:29:17.212654 14370 ts_tablet_manager.cc:1397] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20260812 02:29:17.214550 14373 tablet_bootstrap.cc:654] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:17.214288 14370 raft_consensus.cc:348] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.214727 14370 raft_consensus.cc:374] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:17.215129 14370 raft_consensus.cc:729] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a79f429239646f1a07249872349f7f3, State: Initialized, Role: FOLLOWER
I20260812 02:29:17.212682 14369 raft_consensus.cc:348] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.215373 14369 raft_consensus.cc:374] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:17.215648 14369 raft_consensus.cc:729] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ba67f5b891422faa5ec24c09e626dc, State: Initialized, Role: FOLLOWER
I20260812 02:29:17.215857 14370 consensus_queue.cc:260] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.216084 14369 consensus_queue.cc:260] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.217206 14369 ts_tablet_manager.cc:1428] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260812 02:29:17.217995 14370 ts_tablet_manager.cc:1428] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260812 02:29:17.225646 14373 tablet_bootstrap.cc:492] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: No bootstrap required, opened a new log
I20260812 02:29:17.225929 14373 ts_tablet_manager.cc:1397] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.010s
I20260812 02:29:17.227082 14373 raft_consensus.cc:348] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.227385 14373 raft_consensus.cc:374] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:17.227656 14373 raft_consensus.cc:729] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669dd0b780f4514a6a96413dd5a4234, State: Initialized, Role: FOLLOWER
I20260812 02:29:17.228003 14373 consensus_queue.cc:260] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.228979 14373 ts_tablet_manager.cc:1428] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:29:17.290426 14367 ts_tablet_manager.cc:1918] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:29:17.290865 14367 log.cc:1196] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/954ecc61d5ae41eabf302d79de2ff0d8
I20260812 02:29:17.291832 14367 ts_tablet_manager.cc:1939] T 954ecc61d5ae41eabf302d79de2ff0d8 P 24ba67f5b891422faa5ec24c09e626dc: Deleting consensus metadata
I20260812 02:29:17.291821 14366 ts_tablet_manager.cc:1918] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:29:17.292328 14366 log.cc:1196] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/954ecc61d5ae41eabf302d79de2ff0d8
I20260812 02:29:17.294068 14366 ts_tablet_manager.cc:1939] T 954ecc61d5ae41eabf302d79de2ff0d8 P 4669dd0b780f4514a6a96413dd5a4234: Deleting consensus metadata
I20260812 02:29:17.365121 13681 catalog_manager.cc:4841] TS 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193:44411): tablet 954ecc61d5ae41eabf302d79de2ff0d8 (table test-table [id=feb4728c3ccd4a66ba08ae9d3b361710]) successfully deleted
I20260812 02:29:17.380535 13678 catalog_manager.cc:4841] TS 4669dd0b780f4514a6a96413dd5a4234 (127.12.8.194:42035): tablet 954ecc61d5ae41eabf302d79de2ff0d8 (table test-table [id=feb4728c3ccd4a66ba08ae9d3b361710]) successfully deleted
I20260812 02:29:17.390026 14368 ts_tablet_manager.cc:1918] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:29:17.390389 14368 log.cc:1196] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/954ecc61d5ae41eabf302d79de2ff0d8
I20260812 02:29:17.391297 14368 ts_tablet_manager.cc:1939] T 954ecc61d5ae41eabf302d79de2ff0d8 P 2a79f429239646f1a07249872349f7f3: Deleting consensus metadata
I20260812 02:29:17.394065 13682 catalog_manager.cc:4841] TS 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117): tablet 954ecc61d5ae41eabf302d79de2ff0d8 (table test-table [id=feb4728c3ccd4a66ba08ae9d3b361710]) successfully deleted
I20260812 02:29:17.417807 14359 raft_consensus.cc:482] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:17.418006 14350 raft_consensus.cc:482] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:17.418156 14359 raft_consensus.cc:504] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.418270 14350 raft_consensus.cc:504] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.419240 14359 leader_election.cc:288] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193:44411), 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:17.420583 14350 leader_election.cc:288] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4669dd0b780f4514a6a96413dd5a4234 (127.12.8.194:42035), 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:17.420648 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e70ee08d3e46dd99da4b3a6c21e097" candidate_uuid: "4669dd0b780f4514a6a96413dd5a4234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ba67f5b891422faa5ec24c09e626dc" is_pre_election: true
I20260812 02:29:17.421025 13887 raft_consensus.cc:2454] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4669dd0b780f4514a6a96413dd5a4234 in term 0.
I20260812 02:29:17.423456 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e70ee08d3e46dd99da4b3a6c21e097" candidate_uuid: "4669dd0b780f4514a6a96413dd5a4234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a79f429239646f1a07249872349f7f3" is_pre_election: true
I20260812 02:29:17.423991 14243 raft_consensus.cc:2454] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4669dd0b780f4514a6a96413dd5a4234 in term 0.
I20260812 02:29:17.424929 13977 leader_election.cc:302] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ba67f5b891422faa5ec24c09e626dc, 4669dd0b780f4514a6a96413dd5a4234; no voters:
I20260812 02:29:17.426584 14359 raft_consensus.cc:2781] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:17.426797 14359 raft_consensus.cc:482] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:17.426932 14359 raft_consensus.cc:3037] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:17.443509 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.421250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:33117 (local address 127.12.8.193:55479)
0812 02:29:17.431117 (+ 9867us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:17.431140 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:17.431339 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:17.432150 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:17.432156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:17.432188 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:17.433606 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.433631 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.436001 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.436008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.439292 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.439322 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.442051 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.442059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.442350 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.442392 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 02:29:17.442497 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 02:29:17.442691 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9672,"thread_start_us":17035,"threads_started":1}
I20260812 02:29:17.444782 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.421751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:55479 (local address 127.12.8.195:33117)
0812 02:29:17.430224 (+ 8473us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:17.430231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:17.431538 (+ 1307us) server_negotiation.cc:407] Connection header received
0812 02:29:17.431622 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:17.431626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:17.431702 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:17.431797 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:17.433946 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.435689 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.439672 (+ 3983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.441546 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.441712 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.441803 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:17.444029 (+ 2226us) server_negotiation.cc:299] Negotiation successful
0812 02:29:17.444258 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8321,"thread_start_us":8080,"threads_started":1}
I20260812 02:29:17.445915 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e70ee08d3e46dd99da4b3a6c21e097" candidate_uuid: "24ba67f5b891422faa5ec24c09e626dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a79f429239646f1a07249872349f7f3" is_pre_election: true
I20260812 02:29:17.446374 14243 raft_consensus.cc:2454] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ba67f5b891422faa5ec24c09e626dc in term 0.
I20260812 02:29:17.447219 13808 leader_election.cc:302] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ba67f5b891422faa5ec24c09e626dc, 2a79f429239646f1a07249872349f7f3; no voters:
I20260812 02:29:17.447999 14350 raft_consensus.cc:2781] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:17.448273 14350 raft_consensus.cc:482] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:17.448668 14350 raft_consensus.cc:3037] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:17.449956 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.426139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:35805 (local address 127.12.8.194:42035)
0812 02:29:17.426387 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:17.426396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:17.438269 (+ 11873us) server_negotiation.cc:407] Connection header received
0812 02:29:17.438724 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:17.438731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:17.438821 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:17.438877 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:17.440578 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.442312 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.446381 (+ 4069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.448204 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.448435 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.448519 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:17.449373 (+ 854us) server_negotiation.cc:299] Negotiation successful
0812 02:29:17.449524 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:29:17.450321 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.421964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42035 (local address 127.12.8.193:35805)
0812 02:29:17.438136 (+ 16172us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:17.438158 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:17.438566 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:17.439123 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:17.439133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:17.439158 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:17.440291 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.440316 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.442504 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.442511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.446170 (+ 3659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.446187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.448705 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.448716 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.449104 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.449153 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 02:29:17.449317 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 02:29:17.449506 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16017,"thread_start_us":10391,"threads_started":1}
I20260812 02:29:17.451939 14070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e70ee08d3e46dd99da4b3a6c21e097" candidate_uuid: "24ba67f5b891422faa5ec24c09e626dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4669dd0b780f4514a6a96413dd5a4234" is_pre_election: true
I20260812 02:29:17.497212 14359 raft_consensus.cc:504] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.497236 14350 raft_consensus.cc:504] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.499370 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e70ee08d3e46dd99da4b3a6c21e097" candidate_uuid: "4669dd0b780f4514a6a96413dd5a4234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ba67f5b891422faa5ec24c09e626dc"
I20260812 02:29:17.500002 13887 raft_consensus.cc:2379] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4669dd0b780f4514a6a96413dd5a4234 in current term 1: Already voted for candidate 24ba67f5b891422faa5ec24c09e626dc in this term.
I20260812 02:29:17.499475 14350 leader_election.cc:288] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [CANDIDATE]: Term 1 election: Requested vote from peers 4669dd0b780f4514a6a96413dd5a4234 (127.12.8.194:42035), 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:17.500298 14069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e70ee08d3e46dd99da4b3a6c21e097" candidate_uuid: "24ba67f5b891422faa5ec24c09e626dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4669dd0b780f4514a6a96413dd5a4234"
I20260812 02:29:17.501657 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e70ee08d3e46dd99da4b3a6c21e097" candidate_uuid: "24ba67f5b891422faa5ec24c09e626dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a79f429239646f1a07249872349f7f3"
I20260812 02:29:17.502292 14359 leader_election.cc:288] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [CANDIDATE]: Term 1 election: Requested vote from peers 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193:44411), 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:17.502604 14070 raft_consensus.cc:2379] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 24ba67f5b891422faa5ec24c09e626dc in current term 1: Already voted for candidate 4669dd0b780f4514a6a96413dd5a4234 in this term.
I20260812 02:29:17.502632 14243 raft_consensus.cc:3037] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:17.502453 14242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e70ee08d3e46dd99da4b3a6c21e097" candidate_uuid: "4669dd0b780f4514a6a96413dd5a4234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a79f429239646f1a07249872349f7f3"
I20260812 02:29:17.505280 13978 leader_election.cc:302] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4669dd0b780f4514a6a96413dd5a4234; no voters: 24ba67f5b891422faa5ec24c09e626dc, 2a79f429239646f1a07249872349f7f3
I20260812 02:29:17.505918 14359 raft_consensus.cc:2726] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 02:29:17.605664 14243 raft_consensus.cc:2454] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ba67f5b891422faa5ec24c09e626dc in term 1.
I20260812 02:29:17.606691 13808 leader_election.cc:302] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24ba67f5b891422faa5ec24c09e626dc, 2a79f429239646f1a07249872349f7f3; no voters: 4669dd0b780f4514a6a96413dd5a4234
I20260812 02:29:17.607272 14350 raft_consensus.cc:2781] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:17.608646 14350 raft_consensus.cc:686] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 1 LEADER]: Becoming Leader. State: Replica: 24ba67f5b891422faa5ec24c09e626dc, State: Running, Role: LEADER
I20260812 02:29:17.609306 14350 consensus_queue.cc:237] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.615233 13698 catalog_manager.cc:5495] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193). New cstate: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:29:17 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501754, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:17.715687 14391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.685818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42225 (local address 127.0.0.1:51356)
0812 02:29:17.687746 (+ 1928us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:17.687766 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:17.687964 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:17.688294 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:17.688299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:17.689884 (+ 1585us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:17.690411 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.690425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.691901 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.691905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.693706 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.693719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.695092 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.695104 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.695354 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.699905 (+ 4551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:17.699924 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:17.704117 (+ 4193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:17.711837 (+ 7720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:17.711856 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:17.711863 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:17.712546 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:17.713323 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:17.713326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:17.713329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:17.713501 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:17.714438 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:17.714445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:17.714659 (+ 214us) client_negotiation.cc:769] Sending connection context
0812 02:29:17.714874 (+ 215us) client_negotiation.cc:240] Negotiation successful
0812 02:29:17.715204 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1716,"thread_start_us":1410,"threads_started":1}
I20260812 02:29:17.716218 14390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.685499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.12.8.254:42225)
0812 02:29:17.687236 (+ 1737us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:17.687245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:17.687876 (+ 631us) server_negotiation.cc:407] Connection header received
0812 02:29:17.688081 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:17.688086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:17.688146 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:17.688237 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:17.690625 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.691706 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.693968 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.694834 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.694957 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.704847 (+ 9890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:17.704862 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:17.704869 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:17.704939 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:17.711572 (+ 6633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:17.712762 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:17.712769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:17.712772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:17.713107 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:17.713662 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:17.713668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:17.713673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:17.714273 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:17.714460 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:17.715108 (+ 648us) server_negotiation.cc:299] Negotiation successful
0812 02:29:17.715449 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1473,"thread_start_us":1171,"threads_started":1}
I20260812 02:29:17.724884 13698 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51356:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:29:17.726788 13698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:17.735812 13861 tablet_service.cc:1467] Processing CreateTablet for tablet 50a89c93cd624400bb68d2e5742f90f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8a6a0a649964cc68fc49d578134f21d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:29:17.735978 14215 tablet_service.cc:1467] Processing CreateTablet for tablet 50a89c93cd624400bb68d2e5742f90f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8a6a0a649964cc68fc49d578134f21d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:29:17.736093 14047 tablet_service.cc:1467] Processing CreateTablet for tablet 50a89c93cd624400bb68d2e5742f90f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8a6a0a649964cc68fc49d578134f21d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:29:17.736467 13861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a89c93cd624400bb68d2e5742f90f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:17.736563 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a89c93cd624400bb68d2e5742f90f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:17.736791 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a89c93cd624400bb68d2e5742f90f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:17.753153 14393 tablet_bootstrap.cc:492] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: Bootstrap starting.
I20260812 02:29:17.755127 14392 tablet_bootstrap.cc:492] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: Bootstrap starting.
I20260812 02:29:17.756034 14394 tablet_bootstrap.cc:492] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: Bootstrap starting.
I20260812 02:29:17.758656 14392 tablet_bootstrap.cc:654] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:17.759397 14393 tablet_bootstrap.cc:654] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:17.760344 14394 tablet_bootstrap.cc:654] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:17.763077 14392 tablet_bootstrap.cc:492] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: No bootstrap required, opened a new log
I20260812 02:29:17.763419 14392 ts_tablet_manager.cc:1397] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 02:29:17.766324 14392 raft_consensus.cc:348] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.766992 14392 raft_consensus.cc:374] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:17.767241 14392 raft_consensus.cc:729] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ba67f5b891422faa5ec24c09e626dc, State: Initialized, Role: FOLLOWER
I20260812 02:29:17.767697 14392 consensus_queue.cc:260] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.768396 14350 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.768805 14350 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:17.770526 14394 tablet_bootstrap.cc:492] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: No bootstrap required, opened a new log
I20260812 02:29:17.770862 14394 ts_tablet_manager.cc:1397] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260812 02:29:17.772904 14392 ts_tablet_manager.cc:1428] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260812 02:29:17.773888 14394 raft_consensus.cc:348] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.774362 14394 raft_consensus.cc:374] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:17.774557 14394 raft_consensus.cc:729] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669dd0b780f4514a6a96413dd5a4234, State: Initialized, Role: FOLLOWER
I20260812 02:29:17.774865 14394 consensus_queue.cc:260] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.775013 14393 tablet_bootstrap.cc:492] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: No bootstrap required, opened a new log
I20260812 02:29:17.775211 14393 ts_tablet_manager.cc:1397] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.010s
I20260812 02:29:17.775393 14359 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.775960 14359 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:17.776563 14393 raft_consensus.cc:348] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.776949 14393 raft_consensus.cc:374] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:17.777158 14393 raft_consensus.cc:729] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a79f429239646f1a07249872349f7f3, State: Initialized, Role: FOLLOWER
I20260812 02:29:17.777745 14393 consensus_queue.cc:260] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.779721 14394 ts_tablet_manager.cc:1428] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260812 02:29:17.783432 14393 ts_tablet_manager.cc:1428] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: Time spent starting tablet: real 0.008s user 0.001s sys 0.008s
I20260812 02:29:17.783934 14398 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.784667 14398 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:17.789381 14350 raft_consensus.cc:482] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:17.789691 14350 raft_consensus.cc:504] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.790490 14350 leader_election.cc:288] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4669dd0b780f4514a6a96413dd5a4234 (127.12.8.194:42035), 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:17.791126 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a89c93cd624400bb68d2e5742f90f6" candidate_uuid: "24ba67f5b891422faa5ec24c09e626dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a79f429239646f1a07249872349f7f3" is_pre_election: true
I20260812 02:29:17.791226 14070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a89c93cd624400bb68d2e5742f90f6" candidate_uuid: "24ba67f5b891422faa5ec24c09e626dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4669dd0b780f4514a6a96413dd5a4234" is_pre_election: true
I20260812 02:29:17.791582 14243 raft_consensus.cc:2454] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ba67f5b891422faa5ec24c09e626dc in term 0.
I20260812 02:29:17.791581 14070 raft_consensus.cc:2454] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ba67f5b891422faa5ec24c09e626dc in term 0.
I20260812 02:29:17.792156 13808 leader_election.cc:302] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ba67f5b891422faa5ec24c09e626dc, 2a79f429239646f1a07249872349f7f3; no voters:
I20260812 02:29:17.792713 14350 raft_consensus.cc:2781] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:17.792886 14350 raft_consensus.cc:482] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:17.793087 14350 raft_consensus.cc:3037] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:17.806332 14350 raft_consensus.cc:504] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.807339 14350 leader_election.cc:288] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [CANDIDATE]: Term 1 election: Requested vote from peers 4669dd0b780f4514a6a96413dd5a4234 (127.12.8.194:42035), 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117)
I20260812 02:29:17.807924 14070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a89c93cd624400bb68d2e5742f90f6" candidate_uuid: "24ba67f5b891422faa5ec24c09e626dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4669dd0b780f4514a6a96413dd5a4234"
I20260812 02:29:17.808106 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a89c93cd624400bb68d2e5742f90f6" candidate_uuid: "24ba67f5b891422faa5ec24c09e626dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a79f429239646f1a07249872349f7f3"
I20260812 02:29:17.808331 14070 raft_consensus.cc:3037] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:17.808456 14243 raft_consensus.cc:3037] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:17.812891 14243 raft_consensus.cc:2454] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ba67f5b891422faa5ec24c09e626dc in term 1.
I20260812 02:29:17.813553 13808 leader_election.cc:302] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ba67f5b891422faa5ec24c09e626dc, 2a79f429239646f1a07249872349f7f3; no voters:
I20260812 02:29:17.813898 14350 raft_consensus.cc:2781] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:17.814139 14350 raft_consensus.cc:686] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [term 1 LEADER]: Becoming Leader. State: Replica: 24ba67f5b891422faa5ec24c09e626dc, State: Running, Role: LEADER
I20260812 02:29:17.814554 14350 consensus_queue.cc:237] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } }
I20260812 02:29:17.817330 14380 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: TxnStatusTablet state changed. Reason: New leader 24ba67f5b891422faa5ec24c09e626dc. Latest consensus state: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.817945 14380 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:29:17.820542 14402 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:29:17.820317 13700 catalog_manager.cc:5495] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193). New cstate: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } health_report { overall_health: UNKNOWN } } }
I20260812 02:29:17.820909 14402 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:29:17.863634 14391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.851095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:44411 (local address 127.0.0.1:41974)
0812 02:29:17.851410 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:17.851449 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:17.851663 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:17.853423 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:17.853428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:17.853442 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:29:17.853826 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.853843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.856320 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.856326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.858351 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.858371 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.859935 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.859941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.860431 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.860566 (+ 135us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:29:17.862705 (+ 2139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:29:17.862737 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:29:17.862857 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 02:29:17.863089 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 02:29:17.863857 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.850929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.12.8.193:44411)
0812 02:29:17.852754 (+ 1825us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:17.852760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:17.852810 (+ 50us) server_negotiation.cc:407] Connection header received
0812 02:29:17.852933 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:17.852939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:17.853099 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:17.853338 (+ 239us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:17.854163 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.856088 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.858527 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.859717 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.859883 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.861312 (+ 1429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:17.862544 (+ 1232us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:17.862729 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:17.862920 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 02:29:17.863035 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1599,"thread_start_us":1167,"threads_started":1}
I20260812 02:29:17.873719 14243 raft_consensus.cc:1261] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Refusing update from remote peer 24ba67f5b891422faa5ec24c09e626dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:17.874707 14380 consensus_queue.cc:1035] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:17.882243 14398 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: TxnStatusTablet state changed. Reason: New leader 24ba67f5b891422faa5ec24c09e626dc. Latest consensus state: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.882728 14398 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:17.885900 14407 mvcc.cc:205] Tried to move back new op lower bound from 7317511200240041984 to 7317511200012189696. Current Snapshot: MvccSnapshot[applied={T|T < 7317511200240041984}]
I20260812 02:29:17.894827 14350 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.895538 14350 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:29:17.895520 14398 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.895865 14398 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 2a79f429239646f1a07249872349f7f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:17.906996 14070 raft_consensus.cc:2454] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ba67f5b891422faa5ec24c09e626dc in term 1.
I20260812 02:29:17.908438 14069 raft_consensus.cc:1261] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Refusing update from remote peer 24ba67f5b891422faa5ec24c09e626dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:17.910066 14350 consensus_queue.cc:1035] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:17.941788 14359 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: TxnStatusTablet state changed. Reason: New leader 24ba67f5b891422faa5ec24c09e626dc. Latest consensus state: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.942384 14359 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:17.944087 14416 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.944840 14416 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 4669dd0b780f4514a6a96413dd5a4234: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:17.953248 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.932828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.12.8.193:44411)
0812 02:29:17.933045 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:17.933053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:17.943096 (+ 10043us) server_negotiation.cc:407] Connection header received
0812 02:29:17.945497 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:17.945503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:17.945593 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:17.945652 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:17.946741 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.948150 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.950523 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.951506 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.951616 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.952372 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:17.952527 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:17.952627 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:17.952842 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 02:29:17.952933 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 02:29:17.980928 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.969662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:38087 (local address 127.12.8.193:44411)
0812 02:29:17.969883 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:17.969890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:17.970637 (+ 747us) server_negotiation.cc:407] Connection header received
0812 02:29:17.970922 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:17.970928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:17.972344 (+ 1416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:17.972460 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:17.973615 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.975185 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.978103 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.979005 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.979131 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.979828 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:17.979978 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:17.980080 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:17.980293 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 02:29:17.980416 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":63}
I20260812 02:29:17.981307 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:17.969069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:44411 (local address 127.12.8.193:38087)
0812 02:29:17.970509 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:17.970530 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:17.970743 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:17.972488 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:17.972493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:17.972507 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:29:17.973386 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.973402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.975372 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.975378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.977934 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:17.977951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:17.979214 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:17.979221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:17.979458 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:17.979541 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:29:17.980119 (+ 578us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:29:17.980140 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:29:17.980246 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 02:29:17.980432 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":1,"thread_start_us":857,"threads_started":1}
I20260812 02:29:17.985881 14350 tablet_replica.cc:439] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24ba67f5b891422faa5ec24c09e626dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ba67f5b891422faa5ec24c09e626dc" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 44411 } } peers { permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 } } peers { permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 } } }
I20260812 02:29:17.986548 14350 tablet_replica.cc:442] T 50a89c93cd624400bb68d2e5742f90f6 P 24ba67f5b891422faa5ec24c09e626dc: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:29:17.999073 14243 raft_consensus.cc:1261] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Refusing update from remote peer 24ba67f5b891422faa5ec24c09e626dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:17.999131 14069 raft_consensus.cc:1261] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Refusing update from remote peer 24ba67f5b891422faa5ec24c09e626dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:17.999799 14350 consensus_queue.cc:1035] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a79f429239646f1a07249872349f7f3" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:18.000185 14380 consensus_queue.cc:1035] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4669dd0b780f4514a6a96413dd5a4234" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:18.106922 13698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:18.107344 13698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:18.111306 13698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 355d9148a8ee411aaed4d0bb25e5de81: Sending DeleteTablet for 3 replicas of tablet 46e70ee08d3e46dd99da4b3a6c21e097
I20260812 02:29:18.112294 13863 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e70ee08d3e46dd99da4b3a6c21e097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:18 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:41972
I20260812 02:29:18.112598 14047 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e70ee08d3e46dd99da4b3a6c21e097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:18 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:50210
I20260812 02:29:18.113806 14215 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e70ee08d3e46dd99da4b3a6c21e097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:18 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51160
I20260812 02:29:18.122790 14429 tablet_replica.cc:330] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: stopping tablet replica
I20260812 02:29:18.124485 14427 tablet_replica.cc:330] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc: stopping tablet replica
I20260812 02:29:18.124734 14427 raft_consensus.cc:2229] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:29:18.123088 14429 raft_consensus.cc:2229] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:18.125684 14427 raft_consensus.cc:2258] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:18.125941 14429 raft_consensus.cc:2258] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:18.126943 14428 tablet_replica.cc:330] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: stopping tablet replica
I20260812 02:29:18.128898 14428 raft_consensus.cc:2229] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:18.129369 14428 raft_consensus.cc:2258] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:18.128288 14429 ts_tablet_manager.cc:1905] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:18.129681 14427 ts_tablet_manager.cc:1905] T 46e70ee08d3e46dd99da4b3a6c21e097 P 24ba67f5b891422faa5ec24c09e626dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:18.135455 14428 ts_tablet_manager.cc:1905] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 02:29:18 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:18.160502 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.127078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.12.8.193:44411)
0812 02:29:18.127297 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:18.127303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:18.136008 (+ 8705us) server_negotiation.cc:407] Connection header received
0812 02:29:18.136287 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:18.136293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:18.136380 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:18.136544 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:18.140130 (+ 3586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.141549 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.143338 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.144531 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.144671 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:18.151368 (+ 6697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:18.151386 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:18.151403 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:18.151446 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:18.156383 (+ 4937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.157668 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.157676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.157678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.157894 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.158391 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.158395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.158397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.159030 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:18.159554 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:18.159752 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 02:29:18.159934 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 02:29:18.161748 13947 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41986
I20260812 02:29:18.184957 14390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.165892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.12.8.254:42225)
0812 02:29:18.166149 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:18.166157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:18.166312 (+ 155us) server_negotiation.cc:407] Connection header received
0812 02:29:18.166469 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:18.166475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:18.166559 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:18.166666 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:18.168700 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.170227 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.172022 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.173063 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.173244 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:18.176768 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:18.176776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:18.176783 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:18.176826 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:18.181675 (+ 4849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.182377 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.182383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.182387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.182586 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.183116 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.183120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.183122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.183508 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:18.183850 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:18.184218 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 02:29:18.184515 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260812 02:29:18.185850 13698 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51370
I20260812 02:29:18.188769 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 13784
I20260812 02:29:18.206938 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.189613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.12.8.193:44411)
0812 02:29:18.189788 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:18.189794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:18.190163 (+ 369us) server_negotiation.cc:407] Connection header received
0812 02:29:18.190273 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:18.190278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:18.190378 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:18.190496 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:18.191627 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.192996 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.194252 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.195194 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.195336 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:18.198442 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:18.198452 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:18.198457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:18.198498 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:18.203434 (+ 4936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.204337 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.204343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.204346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.204607 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.205099 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.205104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.205108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.205638 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:18.206061 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:18.206249 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 02:29:18.206496 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 02:29:18.207659 13947 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41990
W20260812 02:29:18.208248 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 13784: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:18.208536 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 13784
I20260812 02:29:18.228389 14428 ts_tablet_manager.cc:1918] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:29:18.228629 14428 log.cc:1196] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/46e70ee08d3e46dd99da4b3a6c21e097
I20260812 02:29:18.228726 14429 ts_tablet_manager.cc:1918] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:29:18.228979 14429 log.cc:1196] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/46e70ee08d3e46dd99da4b3a6c21e097
I20260812 02:29:18.229389 14428 ts_tablet_manager.cc:1939] T 46e70ee08d3e46dd99da4b3a6c21e097 P 4669dd0b780f4514a6a96413dd5a4234: Deleting consensus metadata
I20260812 02:29:18.229800 14429 ts_tablet_manager.cc:1939] T 46e70ee08d3e46dd99da4b3a6c21e097 P 2a79f429239646f1a07249872349f7f3: Deleting consensus metadata
W20260812 02:29:18.245843 13681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.8.193:44411 (error 108)
I20260812 02:29:18.246240 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 13957
I20260812 02:29:18.249454 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.247728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:44411 (local address 127.0.0.1:41992)
0812 02:29:18.248791 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:18.249078 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":768,"threads_started":1}
W20260812 02:29:18.249679 13681 catalog_manager.cc:4584] TS 24ba67f5b891422faa5ec24c09e626dc (127.12.8.193:44411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46e70ee08d3e46dd99da4b3a6c21e097: Network error: Client connection negotiation failed: client connection to 127.12.8.193:44411: connect: Connection refused (error 111)
Aug 12 02:29:18 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:29:18.262580 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.247066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.12.8.194:42035)
0812 02:29:18.247561 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:18.247567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:18.247666 (+ 99us) server_negotiation.cc:407] Connection header received
0812 02:29:18.247803 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:18.247808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:18.247908 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:18.248042 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:18.249709 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.250508 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.251752 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.252210 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.252335 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:18.256732 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:18.256764 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:18.256781 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:18.256826 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:18.259760 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.260524 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.260532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.260534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.260677 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.261033 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.261038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.261041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.261507 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:18.261737 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:18.262065 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 02:29:18.262245 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":344,"threads_started":1}
W20260812 02:29:18.263304 14113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50234
W20260812 02:29:18.263935 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 13957: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:18.264101 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 13957
I20260812 02:29:18.298970 13682 catalog_manager.cc:4841] TS 2a79f429239646f1a07249872349f7f3 (127.12.8.195:33117): tablet 46e70ee08d3e46dd99da4b3a6c21e097 (table test-table [id=414a647dd984479295c5e47386886e11]) successfully deleted
I20260812 02:29:18.303586 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.303060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:44411 (local address 127.0.0.1:41996)
0812 02:29:18.303295 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:18.303413 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 02:29:18.304829 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.304404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:44411 (local address 127.0.0.1:41998)
0812 02:29:18.304560 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:18.304662 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 02:29:18.315897 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14126
I20260812 02:29:18.317008 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.316455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42035 (local address 127.0.0.1:50250)
0812 02:29:18.316736 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:18.316830 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.194:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
Aug 12 02:29:18 dist-test-slave-vq2q krb5kdc[13645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:29:18.332441 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.316635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.12.8.195:33117)
0812 02:29:18.317562 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:18.317566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:18.317582 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:29:18.317634 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:18.317637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:18.317714 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:18.317841 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:18.319118 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.319940 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.321219 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:18.321656 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:18.321765 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:18.326286 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:18.326304 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:18.326321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:18.326355 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:18.329595 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.330330 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.330339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.330342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.330489 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:18.330827 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:18.330830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:18.330833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:18.331225 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:18.331524 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:18.331838 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 02:29:18.332124 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":688,"threads_started":1}
W20260812 02:29:18.333132 14295 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51172
W20260812 02:29:18.333853 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14126: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:18.333997 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 14126
I20260812 02:29:18.349655 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.349071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42035 (local address 127.0.0.1:50266)
0812 02:29:18.349332 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:18.349436 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.194:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 02:29:18.350677 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.350275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42035 (local address 127.0.0.1:50280)
0812 02:29:18.350456 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:18.350550 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.194:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 02:29:18.356254 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 13659
I20260812 02:29:18.357304 13747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:18.476725 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:18.475855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:44411 (local address 127.0.0.1:42014)
0812 02:29:18.476351 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:18.476477 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368}
I20260812 02:29:18.476902 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 13659
2026-08-12T02:29:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6007 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:18 dist-test-slave-vq2q krb5kdc[14445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:18 dist-test-slave-vq2q krb5kdc[14445](info): set up 2 sockets
Aug 12 02:29:18 dist-test-slave-vq2q krb5kdc[14445](info): commencing operation
krb5kdc: starting...
W20260812 02:29:19.590749 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:19 dist-test-slave-vq2q krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:19.882017 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:44955
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:44955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:20.311244 14464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:20.311664 14464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:20.311813 14464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:20.320128 14464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:20.320307 14464 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:29:20.320358 14464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:20.320462 14464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:20.320540 14464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:20.320641 14464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:20.332232 14464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:44955
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:44955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:20.333995 14464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:20.335633 14464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:20.350572 14469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:20.350534 14473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:20.350591 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:20.351325 14464 server_base.cc:1034] running on GCE node
I20260812 02:29:20.352416 14464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:20.354413 14464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:20.355697 14464 hybrid_clock.cc:648] HybridClock initialized: now 1786501760355617 us; error 47 us; skew 500 ppm
I20260812 02:29:20.356104 14464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:20 dist-test-slave-vq2q krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501760, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:20.359894 14464 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:20.361879 14464 webserver.cc:460] Webserver started at http://127.12.8.254:34721/ using document root <none> and password file <none>
I20260812 02:29:20.362811 14464 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:20.362926 14464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:20.363463 14464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:20.367381 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "f93c3ad5769948cfa73037457ef29fe7"
format_stamp: "Formatted at 2026-08-12 02:29:20 on dist-test-slave-vq2q"
server_key: "5d29b7c1456f26929bf35152a8e95b3f"
server_key_iv: "00b2d86807318329e075b9ee4fbcfd35"
server_key_version: "encryptionkey@0"
I20260812 02:29:20.368378 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "f93c3ad5769948cfa73037457ef29fe7"
format_stamp: "Formatted at 2026-08-12 02:29:20 on dist-test-slave-vq2q"
server_key: "5d29b7c1456f26929bf35152a8e95b3f"
server_key_iv: "00b2d86807318329e075b9ee4fbcfd35"
server_key_version: "encryptionkey@0"
I20260812 02:29:20.375229 14464 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260812 02:29:20.379702 14482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:20.381606 14464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:29:20.381830 14464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "f93c3ad5769948cfa73037457ef29fe7"
format_stamp: "Formatted at 2026-08-12 02:29:20 on dist-test-slave-vq2q"
server_key: "5d29b7c1456f26929bf35152a8e95b3f"
server_key_iv: "00b2d86807318329e075b9ee4fbcfd35"
server_key_version: "encryptionkey@0"
I20260812 02:29:20.382099 14464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:20.427870 14464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:20.433698 14464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:20.434034 14464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:20.459842 14464 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:44955
I20260812 02:29:20.459873 14557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:44955 every 8 connection(s)
I20260812 02:29:20.461630 14464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:20.466197 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 14464
I20260812 02:29:20.466500 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:20.467600 12323 external_mini_cluster.cc:1428] Setting key 77039deb6f450cb8b1d97b7882c37115
I20260812 02:29:20.467690 14558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 02:29:20 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:20.485169 14558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: Bootstrap starting.
I20260812 02:29:20.490479 14558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:20.492070 14558 log.cc:824] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:20.496367 14558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: No bootstrap required, opened a new log
I20260812 02:29:20.496837 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:20.470240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.12.8.254:44955)
0812 02:29:20.471216 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:20.471231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:20.471288 (+ 57us) server_negotiation.cc:407] Connection header received
0812 02:29:20.473642 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:20.473673 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:20.474221 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:20.474658 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:20.476808 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:20.478366 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:20.480232 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:20.481046 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:20.481260 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:20.486787 (+ 5527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:20.486800 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:20.486827 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:20.486905 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:20.491342 (+ 4437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:20.492442 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:20.492451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:20.492459 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:20.492695 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:20.493260 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:20.493265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:20.493269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:20.494194 (+ 925us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:20.494466 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:20.494967 (+ 501us) server_negotiation.cc:299] Negotiation successful
0812 02:29:20.495482 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":452,"threads_started":1}
I20260812 02:29:20.505932 14558 raft_consensus.cc:348] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c3ad5769948cfa73037457ef29fe7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44955 } }
I20260812 02:29:20.506625 14558 raft_consensus.cc:374] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:20.506747 14558 raft_consensus.cc:729] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93c3ad5769948cfa73037457ef29fe7, State: Initialized, Role: FOLLOWER
I20260812 02:29:20.507525 14558 consensus_queue.cc:260] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c3ad5769948cfa73037457ef29fe7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44955 } }
I20260812 02:29:20.507825 14558 raft_consensus.cc:388] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:20.507982 14558 raft_consensus.cc:482] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:20.508155 14558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:20.510900 14558 raft_consensus.cc:504] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c3ad5769948cfa73037457ef29fe7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44955 } }
I20260812 02:29:20.511559 14558 leader_election.cc:302] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f93c3ad5769948cfa73037457ef29fe7; no voters:
I20260812 02:29:20.512290 14558 leader_election.cc:288] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:20.512501 14564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:20.513353 14564 raft_consensus.cc:686] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [term 1 LEADER]: Becoming Leader. State: Replica: f93c3ad5769948cfa73037457ef29fe7, State: Running, Role: LEADER
I20260812 02:29:20.514194 14564 consensus_queue.cc:237] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c3ad5769948cfa73037457ef29fe7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44955 } }
I20260812 02:29:20.515192 14558 sys_catalog.cc:564] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:20.517710 14566 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f93c3ad5769948cfa73037457ef29fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c3ad5769948cfa73037457ef29fe7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44955 } } }
I20260812 02:29:20.517846 14567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f93c3ad5769948cfa73037457ef29fe7. Latest consensus state: current_term: 1 leader_uuid: "f93c3ad5769948cfa73037457ef29fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93c3ad5769948cfa73037457ef29fe7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44955 } } }
I20260812 02:29:20.518468 14566 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:20.520354 14567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:20.522251 14579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:20.529446 14579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:20.543278 14579 catalog_manager.cc:1347] Generated new cluster ID: dba7ed1880cb45c9b31415a7d720f376
I20260812 02:29:20.543507 14579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:20.558426 14579 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:20.560259 14579 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:20.568667 14579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: Generated new TSK 0
I20260812 02:29:20.570220 14579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:20.637490 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:44955
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:21.045429 14588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:21.045764 14588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:21.045917 14588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:21.053531 14588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:21.053683 14588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:21.053817 14588 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:29:21.053886 14588 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:29:21.053990 14588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:21.064654 14588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:21.066267 14588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:21.067716 14588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:21.081642 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:21.081856 14596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:21.082042 14594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:21.082772 14588 server_base.cc:1034] running on GCE node
I20260812 02:29:21.083446 14588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:21.084456 14588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:21.085691 14588 hybrid_clock.cc:648] HybridClock initialized: now 1786501761085632 us; error 41 us; skew 500 ppm
I20260812 02:29:21.086048 14588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:21 dist-test-slave-vq2q krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501761, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:21.089442 14588 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:21.090947 14588 webserver.cc:460] Webserver started at http://127.12.8.193:34953/ using document root <none> and password file <none>
I20260812 02:29:21.091684 14588 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:21.091809 14588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:21.092145 14588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:21.095345 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "72504797e7f74fd4b43991a3672fb6e6"
format_stamp: "Formatted at 2026-08-12 02:29:21 on dist-test-slave-vq2q"
server_key: "2d2b5755cc2e9afbdb121d0e74866ae1"
server_key_iv: "78330b62a3eff79e62b2897021b790b4"
server_key_version: "encryptionkey@0"
I20260812 02:29:21.096155 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "72504797e7f74fd4b43991a3672fb6e6"
format_stamp: "Formatted at 2026-08-12 02:29:21 on dist-test-slave-vq2q"
server_key: "2d2b5755cc2e9afbdb121d0e74866ae1"
server_key_iv: "78330b62a3eff79e62b2897021b790b4"
server_key_version: "encryptionkey@0"
I20260812 02:29:21.101898 14588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 02:29:21.105798 14604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:21.107244 14588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:29:21.107420 14588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "72504797e7f74fd4b43991a3672fb6e6"
format_stamp: "Formatted at 2026-08-12 02:29:21 on dist-test-slave-vq2q"
server_key: "2d2b5755cc2e9afbdb121d0e74866ae1"
server_key_iv: "78330b62a3eff79e62b2897021b790b4"
server_key_version: "encryptionkey@0"
I20260812 02:29:21.107636 14588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:21.122973 14588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:21.127368 14588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:21.127671 14588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:21.130618 14588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:21.130725 14588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:21.130861 14588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:21.130941 14588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:29:21 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501761, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:21.154235 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:21.131367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:52759 (local address 127.12.8.254:44955)
0812 02:29:21.131903 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:21.131910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:21.133071 (+ 1161us) server_negotiation.cc:407] Connection header received
0812 02:29:21.135199 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:21.135206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:21.135309 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:21.135438 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:21.138168 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.139145 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.142126 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.142670 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.142816 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:21.147588 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:21.147593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:21.147596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:21.147635 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:21.150468 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:21.151589 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:21.151595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:21.151599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:21.151744 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:21.152217 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:21.152222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:21.152226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:21.152532 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:21.152680 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:21.153531 (+ 851us) server_negotiation.cc:299] Negotiation successful
0812 02:29:21.153801 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":254,"threads_started":1}
I20260812 02:29:21.155942 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:21.131953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44955 (local address 127.12.8.193:52759)
0812 02:29:21.132854 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:21.132923 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:21.134904 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:21.135800 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:21.135813 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:21.136750 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:21.137915 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:21.137952 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.139355 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.139361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:21.141947 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:21.141964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.142956 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.142964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:21.143281 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:21.144229 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:21.144259 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:21.147342 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:21.150672 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:21.150687 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:21.150710 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:21.151423 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:21.151860 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:21.151864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:21.151866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:21.152091 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:21.152675 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:21.152683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:21.153084 (+ 401us) client_negotiation.cc:769] Sending connection context
0812 02:29:21.153398 (+ 314us) client_negotiation.cc:240] Negotiation successful
0812 02:29:21.154043 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":391,"threads_started":1}
I20260812 02:29:21.177867 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:21.163220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44955 (local address 127.12.8.193:37583)
0812 02:29:21.163891 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:21.163914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:21.164042 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:21.164400 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:21.164403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:21.164916 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:21.165274 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:21.165286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.166426 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.166433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:21.168035 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:21.168056 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.168978 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.168985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:21.169143 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:21.169871 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:21.169897 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:21.171104 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:21.174858 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:21.174864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:21.174872 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:21.175481 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:21.175958 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:21.175967 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:21.175971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:21.176161 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:21.176860 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:21.176864 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:21.177039 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 02:29:21.177158 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 02:29:21.177511 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":353,"threads_started":1}
I20260812 02:29:21.177997 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:21.163366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:37583 (local address 127.12.8.254:44955)
0812 02:29:21.163597 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:21.163604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:21.164000 (+ 396us) server_negotiation.cc:407] Connection header received
0812 02:29:21.164155 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:21.164161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:21.164255 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:21.164394 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:21.165423 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.166280 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.168189 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.168785 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.168904 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:21.171387 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:21.171395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:21.171399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:21.171448 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:21.174725 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:21.175657 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:21.175663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:21.175666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:21.175850 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:21.176292 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:21.176298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:21.176303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:21.176735 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:21.176917 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:21.177318 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 02:29:21.177611 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 02:29:21.183692 14588 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:32887
I20260812 02:29:21.183738 14752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:32887 every 8 connection(s)
I20260812 02:29:21.184916 14588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:21.187004 14753 heartbeater.cc:347] Connected to a master server at 127.12.8.254:44955
I20260812 02:29:21.187227 14753 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:21.187917 14753 heartbeater.cc:511] Master 127.12.8.254:44955 requested a full tablet report, sending...
I20260812 02:29:21.189128 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 14588
I20260812 02:29:21.189376 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:21.190243 12323 external_mini_cluster.cc:1428] Setting key 07017d7fe604b0d1f13837245eac40cb
I20260812 02:29:21.190201 14503 ts_manager.cc:194] Registered new tserver with Master: 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887)
I20260812 02:29:21.191908 14503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:52759
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:21.240926 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:44955
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:21.588604 14759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:21.589040 14759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:21.589262 14759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:21.596827 14759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:21.596968 14759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:21.597107 14759 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:29:21.597198 14759 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:29:21.597328 14759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:21.607873 14759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:21.609369 14759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:21.610710 14759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:21.624723 14766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:21.624737 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:21.625276 14765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:21.625682 14759 server_base.cc:1034] running on GCE node
I20260812 02:29:21.626395 14759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:21.627415 14759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:21.628672 14759 hybrid_clock.cc:648] HybridClock initialized: now 1786501761628607 us; error 49 us; skew 500 ppm
I20260812 02:29:21.629050 14759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:21 dist-test-slave-vq2q krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501761, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:21.632514 14759 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:21.634128 14759 webserver.cc:460] Webserver started at http://127.12.8.194:33941/ using document root <none> and password file <none>
I20260812 02:29:21.634820 14759 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:21.634918 14759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:21.635216 14759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:21.638278 14759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "25fd0582c11648059f535cac4b51319a"
format_stamp: "Formatted at 2026-08-12 02:29:21 on dist-test-slave-vq2q"
server_key: "a378dff4b64b12fa041012caaa0fc9d9"
server_key_iv: "609fba63e9e77a0ebd8ee79972c9cb2f"
server_key_version: "encryptionkey@0"
I20260812 02:29:21.638999 14759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "25fd0582c11648059f535cac4b51319a"
format_stamp: "Formatted at 2026-08-12 02:29:21 on dist-test-slave-vq2q"
server_key: "a378dff4b64b12fa041012caaa0fc9d9"
server_key_iv: "609fba63e9e77a0ebd8ee79972c9cb2f"
server_key_version: "encryptionkey@0"
I20260812 02:29:21.644397 14759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:29:21.648209 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:21.649704 14759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:29:21.649848 14759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "25fd0582c11648059f535cac4b51319a"
format_stamp: "Formatted at 2026-08-12 02:29:21 on dist-test-slave-vq2q"
server_key: "a378dff4b64b12fa041012caaa0fc9d9"
server_key_iv: "609fba63e9e77a0ebd8ee79972c9cb2f"
server_key_version: "encryptionkey@0"
I20260812 02:29:21.650046 14759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:21.672258 14759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:21.676397 14759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:21.676681 14759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:21.679633 14759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:21.679731 14759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:21.679848 14759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:21.679932 14759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:29:21 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501761, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:21.703637 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:21.680228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:54903 (local address 127.12.8.254:44955)
0812 02:29:21.681139 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:21.681146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:21.682522 (+ 1376us) server_negotiation.cc:407] Connection header received
0812 02:29:21.684337 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:21.684345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:21.684442 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:21.684568 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:21.686760 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.688058 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.689946 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.690597 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.690737 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:21.695719 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:21.695726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:21.695731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:21.695777 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:21.699282 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:21.700446 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:21.700451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:21.700454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:21.700613 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:21.701306 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:21.701312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:21.701316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:21.701664 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:21.701829 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:21.702976 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0812 02:29:21.703242 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":666,"threads_started":1}
I20260812 02:29:21.704849 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:21.681027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44955 (local address 127.12.8.194:54903)
0812 02:29:21.681987 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:21.682046 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:21.684060 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:21.684837 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:21.684847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:21.685622 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:21.686554 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:21.686585 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.688218 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.688221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:21.689700 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:21.689716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.690876 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.690887 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:21.691133 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:21.692000 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:21.692030 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:21.695398 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:21.699513 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:21.699522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:21.699543 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:21.700239 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:21.700762 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:21.700768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:21.700775 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:21.701135 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:21.701923 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:21.701931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:21.702507 (+ 576us) client_negotiation.cc:769] Sending connection context
0812 02:29:21.702859 (+ 352us) client_negotiation.cc:240] Negotiation successful
0812 02:29:21.703470 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":420,"threads_started":1}
I20260812 02:29:21.726655 14759 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:42809
I20260812 02:29:21.726688 14918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:42809 every 8 connection(s)
I20260812 02:29:21.727335 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:21.711672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:53663 (local address 127.12.8.254:44955)
0812 02:29:21.711874 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:21.711879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:21.712640 (+ 761us) server_negotiation.cc:407] Connection header received
0812 02:29:21.712967 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:21.712973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:21.713066 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:21.713220 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:21.714646 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.715861 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.717317 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.717963 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.718081 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:21.720493 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:21.720500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:21.720504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:21.720548 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:21.724078 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:21.724890 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:21.724895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:21.724898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:21.725058 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:21.725590 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:21.725595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:21.725599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:21.726112 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:21.726279 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:21.726748 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 02:29:21.726979 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 02:29:21.727449 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:21.711765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44955 (local address 127.12.8.194:53663)
0812 02:29:21.712462 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:21.712492 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:21.712799 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:21.713279 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:21.713291 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:21.713886 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:21.714399 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:21.714416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.716026 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.716032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:21.717144 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:21.717158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:21.718155 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:21.718160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:21.718295 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:21.719002 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:21.719030 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:21.720277 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:21.724224 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:21.724230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:21.724234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:21.724768 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:21.725237 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:21.725242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:21.725246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:21.725451 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:21.726263 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:21.726268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:21.726468 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 02:29:21.726599 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 02:29:21.726979 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":370,"threads_started":1}
I20260812 02:29:21.728293 14759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:21.730396 14919 heartbeater.cc:347] Connected to a master server at 127.12.8.254:44955
I20260812 02:29:21.730671 14919 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:21.731410 14919 heartbeater.cc:511] Master 127.12.8.254:44955 requested a full tablet report, sending...
I20260812 02:29:21.732685 14503 ts_manager.cc:194] Registered new tserver with Master: 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809)
I20260812 02:29:21.733537 14503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:54903
I20260812 02:29:21.734185 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 14759
I20260812 02:29:21.734362 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:21.734824 12323 external_mini_cluster.cc:1428] Setting key 8952f5de9c6138d02e3a38e08025e3f3
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:21.783217 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:44955
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:22.120466 14924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:22.120770 14924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:22.120909 14924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:22.128284 14924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:22.128420 14924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:22.128552 14924 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:29:22.128621 14924 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:29:22.128727 14924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:22.138674 14924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:22.139976 14924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:22.141324 14924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:22.155498 14929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:22.155377 14930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:22.155613 14933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:22.156414 14924 server_base.cc:1034] running on GCE node
I20260812 02:29:22.157050 14924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:22.158051 14924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:22.159313 14924 hybrid_clock.cc:648] HybridClock initialized: now 1786501762159265 us; error 43 us; skew 500 ppm
I20260812 02:29:22.159665 14924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:22 dist-test-slave-vq2q krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:22.162952 14924 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:29:22.164558 14924 webserver.cc:460] Webserver started at http://127.12.8.195:41107/ using document root <none> and password file <none>
I20260812 02:29:22.165315 14924 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:22.165450 14924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:22.165776 14924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:22.168704 14924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "ff6aace3e9a743c48b7e766e83bcfc02"
format_stamp: "Formatted at 2026-08-12 02:29:22 on dist-test-slave-vq2q"
server_key: "e528638370d8958e498dcb5a9efab607"
server_key_iv: "ac5ebd2dc9a1ce49020efe3865cd092e"
server_key_version: "encryptionkey@0"
I20260812 02:29:22.169502 14924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "ff6aace3e9a743c48b7e766e83bcfc02"
format_stamp: "Formatted at 2026-08-12 02:29:22 on dist-test-slave-vq2q"
server_key: "e528638370d8958e498dcb5a9efab607"
server_key_iv: "ac5ebd2dc9a1ce49020efe3865cd092e"
server_key_version: "encryptionkey@0"
I20260812 02:29:22.175339 14924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:29:22.179780 14942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:22.181671 14924 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 02:29:22.181928 14924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "ff6aace3e9a743c48b7e766e83bcfc02"
format_stamp: "Formatted at 2026-08-12 02:29:22 on dist-test-slave-vq2q"
server_key: "e528638370d8958e498dcb5a9efab607"
server_key_iv: "ac5ebd2dc9a1ce49020efe3865cd092e"
server_key_version: "encryptionkey@0"
I20260812 02:29:22.182200 14924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:22.195137 14753 heartbeater.cc:503] Master 127.12.8.254:44955 was elected leader, sending a full tablet report...
I20260812 02:29:22.195772 14924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:22.201355 14924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:22.201725 14924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:22.205456 14924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:22.205616 14924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:22.205775 14924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:22.205889 14924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:29:22 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:22.236755 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.206316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:44731 (local address 127.12.8.254:44955)
0812 02:29:22.206533 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.206539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.208525 (+ 1986us) server_negotiation.cc:407] Connection header received
0812 02:29:22.211053 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.211066 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.211209 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.211375 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:22.214056 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.215068 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.219298 (+ 4230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.219966 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.220096 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.226801 (+ 6705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:22.226809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:22.226813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:22.226861 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:22.230999 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:22.232317 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:22.232322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:22.232325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:22.232495 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:22.233708 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:22.233717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:22.233721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:22.234306 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:22.234559 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.235782 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.236168 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 02:29:22.237285 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.207206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44955 (local address 127.12.8.195:44731)
0812 02:29:22.208311 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.208382 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.210535 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.211710 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.211727 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.212699 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:22.213726 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.213759 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.215362 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.215376 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.218997 (+ 3621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.219037 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.220369 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.220380 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.220817 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.222022 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:22.222064 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:22.226503 (+ 4439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:22.231170 (+ 4667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:22.231181 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:22.231202 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:22.232176 (+ 974us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:22.232723 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:22.232736 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:22.232741 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:22.233381 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:22.234540 (+ 1159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:22.234552 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:22.235120 (+ 568us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.235563 (+ 443us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.236051 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":468,"threads_started":1}
I20260812 02:29:22.259382 14924 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:41797
I20260812 02:29:22.259526 15086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:41797 every 8 connection(s)
I20260812 02:29:22.261437 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.244083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44955 (local address 127.12.8.195:45167)
0812 02:29:22.245408 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.245444 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.245719 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.246157 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.246163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.246967 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:22.247481 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.247497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.248592 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.248596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.250011 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.250031 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.250987 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.250995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.251208 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.252100 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:22.252144 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:22.253917 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:22.257533 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:22.257541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:22.257547 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:22.258315 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:22.259062 (+ 747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:22.259069 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:22.259073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:22.259370 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:22.260316 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:22.260325 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:22.260533 (+ 208us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.260677 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.260995 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":782,"threads_started":1}
I20260812 02:29:22.261715 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.244007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:45167 (local address 127.12.8.254:44955)
0812 02:29:22.244206 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.244212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.245589 (+ 1377us) server_negotiation.cc:407] Connection header received
0812 02:29:22.245893 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.245897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.245973 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.246082 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:22.247674 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.248425 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.250165 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.250698 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.250811 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.254234 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:22.254241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:22.254246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:22.254288 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:22.257354 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:22.258614 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:22.258622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:22.258626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:22.258909 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:22.259647 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:22.259653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:22.259657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:22.260150 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:22.260350 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.260882 (+ 532us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.261260 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:29:22.261754 14924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:22.264515 15089 heartbeater.cc:347] Connected to a master server at 127.12.8.254:44955
I20260812 02:29:22.264803 15089 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:22.265933 15089 heartbeater.cc:511] Master 127.12.8.254:44955 requested a full tablet report, sending...
I20260812 02:29:22.267374 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 14924
I20260812 02:29:22.267552 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:22.267815 14503 ts_manager.cc:194] Registered new tserver with Master: ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
I20260812 02:29:22.268013 12323 external_mini_cluster.cc:1428] Setting key cf0249a95af2bfa463a7e170b4d09c2d
I20260812 02:29:22.268940 14503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:44731
I20260812 02:29:22.272058 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:29:22 dist-test-slave-vq2q krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:29:22 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:22.336697 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.322154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.12.8.254:44955)
0812 02:29:22.322444 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.322452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.322590 (+ 138us) server_negotiation.cc:407] Connection header received
0812 02:29:22.322826 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.322832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.322916 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.323007 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:22.324522 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.325332 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.326600 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.327017 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.327125 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.331401 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:22.331406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:22.331409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:22.331443 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:22.334139 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:22.334793 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:22.334798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:22.334800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:22.334923 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:22.335270 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:22.335274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:22.335276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:22.335553 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:22.335853 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.336117 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.336344 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 02:29:22.342350 14503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:22.350653 14503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:22.378715 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.366950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.12.8.195:41797)
0812 02:29:22.368104 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.368115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.368922 (+ 807us) server_negotiation.cc:407] Connection header received
0812 02:29:22.369049 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.369058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.369395 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.369593 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:22.370798 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.372957 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.375969 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.377465 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.377601 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.377864 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.378165 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.378359 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":666,"threads_started":1}
I20260812 02:29:22.378980 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.366166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:41797 (local address 127.0.0.1:46316)
0812 02:29:22.368611 (+ 2445us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.368633 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.368784 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.370033 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.370043 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.370085 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:22.370550 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.370566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.373122 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.373128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.375735 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.375752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.377656 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.377662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.377934 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.377979 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.378111 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.378333 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2288,"mutex_wait_us":76,"spinlock_wait_cycles":18176,"thread_start_us":2129,"threads_started":1}
I20260812 02:29:22.379323 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.367498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42809 (local address 127.0.0.1:33540)
0812 02:29:22.370071 (+ 2573us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.370092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.370311 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.371027 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.371033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.371059 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:22.371538 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.371553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.373212 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.373218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.375631 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.375651 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.378114 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.378121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.378336 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.378372 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.378472 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.378658 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2429,"mutex_wait_us":6,"spinlock_wait_cycles":3584,"thread_start_us":2258,"threads_started":1}
I20260812 02:29:22.379951 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.365735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:32887 (local address 127.0.0.1:51460)
0812 02:29:22.366917 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.366967 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.367199 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.371231 (+ 4032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.371241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.371270 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:22.371933 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.371949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.374171 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.374177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.376599 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.376616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.378629 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.378639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.378849 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.378890 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.378956 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.379104 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":2,"thread_start_us":677,"threads_started":1}
I20260812 02:29:22.380344 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.365966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.12.8.193:32887)
0812 02:29:22.370462 (+ 4496us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.370471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.370500 (+ 29us) server_negotiation.cc:407] Connection header received
0812 02:29:22.370612 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.370619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.370952 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.372183 (+ 1231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:22.372443 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.373985 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.376812 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.378373 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.379451 (+ 1078us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.379689 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.379807 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.379999 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4310,"thread_start_us":4121,"threads_started":1}
I20260812 02:29:22.380990 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.368170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.12.8.194:42809)
0812 02:29:22.369535 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.369548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.370450 (+ 902us) server_negotiation.cc:407] Connection header received
0812 02:29:22.370588 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.370596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.370830 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.371008 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:22.371803 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.373017 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.375891 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.377516 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.377669 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.377934 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.380340 (+ 2406us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.380653 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1158,"thread_start_us":974,"threads_started":1}
I20260812 02:29:22.383185 15003 tablet_service.cc:1467] Processing CreateTablet for tablet 6240da93c8674256b18401546b0679d6 (DEFAULT_TABLE table=test-table [id=0d792fb48a364f2da3805faa1ee4ef7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:22.384512 15003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6240da93c8674256b18401546b0679d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:22.385249 14840 tablet_service.cc:1467] Processing CreateTablet for tablet 6240da93c8674256b18401546b0679d6 (DEFAULT_TABLE table=test-table [id=0d792fb48a364f2da3805faa1ee4ef7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:22.385253 14671 tablet_service.cc:1467] Processing CreateTablet for tablet 6240da93c8674256b18401546b0679d6 (DEFAULT_TABLE table=test-table [id=0d792fb48a364f2da3805faa1ee4ef7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:22.387173 14840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6240da93c8674256b18401546b0679d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:22.388103 14671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6240da93c8674256b18401546b0679d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:22.396399 15112 tablet_bootstrap.cc:492] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Bootstrap starting.
I20260812 02:29:22.399785 15112 tablet_bootstrap.cc:654] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:22.401278 15112 log.cc:824] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:22.404080 15112 tablet_bootstrap.cc:492] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: No bootstrap required, opened a new log
I20260812 02:29:22.404350 15112 ts_tablet_manager.cc:1397] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 02:29:22.405570 15115 tablet_bootstrap.cc:492] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Bootstrap starting.
I20260812 02:29:22.406620 15114 tablet_bootstrap.cc:492] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Bootstrap starting.
I20260812 02:29:22.410710 15114 tablet_bootstrap.cc:654] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:22.412091 15115 tablet_bootstrap.cc:654] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:22.412423 15114 log.cc:824] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:22.413801 15115 log.cc:824] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:22.415304 15112 raft_consensus.cc:348] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.415850 15112 raft_consensus.cc:374] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:22.415961 15112 raft_consensus.cc:729] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6aace3e9a743c48b7e766e83bcfc02, State: Initialized, Role: FOLLOWER
I20260812 02:29:22.416673 15114 tablet_bootstrap.cc:492] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: No bootstrap required, opened a new log
I20260812 02:29:22.416749 15112 consensus_queue.cc:260] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.417009 15114 ts_tablet_manager.cc:1397] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260812 02:29:22.417654 15115 tablet_bootstrap.cc:492] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: No bootstrap required, opened a new log
I20260812 02:29:22.418000 15115 ts_tablet_manager.cc:1397] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260812 02:29:22.427491 15115 raft_consensus.cc:348] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.428040 15115 raft_consensus.cc:374] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:22.428160 15115 raft_consensus.cc:729] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fd0582c11648059f535cac4b51319a, State: Initialized, Role: FOLLOWER
I20260812 02:29:22.429314 15089 heartbeater.cc:503] Master 127.12.8.254:44955 was elected leader, sending a full tablet report...
I20260812 02:29:22.429231 15115 consensus_queue.cc:260] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.429098 15114 raft_consensus.cc:348] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.429648 15114 raft_consensus.cc:374] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:22.429756 15114 raft_consensus.cc:729] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72504797e7f74fd4b43991a3672fb6e6, State: Initialized, Role: FOLLOWER
I20260812 02:29:22.430061 15112 ts_tablet_manager.cc:1428] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Time spent starting tablet: real 0.026s user 0.018s sys 0.008s
I20260812 02:29:22.430595 15114 consensus_queue.cc:260] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.431499 14919 heartbeater.cc:503] Master 127.12.8.254:44955 was elected leader, sending a full tablet report...
I20260812 02:29:22.432232 15115 ts_tablet_manager.cc:1428] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260812 02:29:22.434250 15114 ts_tablet_manager.cc:1428] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
W20260812 02:29:22.437430 14754 tablet.cc:2369] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:22.470997 15122 raft_consensus.cc:482] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:22.471437 15122 raft_consensus.cc:504] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.474119 15122 leader_election.cc:288] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809), ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
W20260812 02:29:22.480562 14920 tablet.cc:2369] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:22.482347 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.474547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:44403 (local address 127.12.8.194:42809)
0812 02:29:22.474840 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.474846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.475921 (+ 1075us) server_negotiation.cc:407] Connection header received
0812 02:29:22.476135 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.476141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.476211 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.476299 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:22.477094 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.477932 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.480044 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.481203 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.481395 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.481474 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.481962 (+ 488us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.482086 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 02:29:22.482344 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.474725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:37467 (local address 127.12.8.195:41797)
0812 02:29:22.475024 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.475031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.475998 (+ 967us) server_negotiation.cc:407] Connection header received
0812 02:29:22.476217 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.476222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.476293 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.476374 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:22.477233 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.477995 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.480206 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.481125 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.481365 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.481454 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.481849 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.482052 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260812 02:29:22.482699 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.474747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:41797 (local address 127.12.8.193:37467)
0812 02:29:22.475867 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.475884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.476065 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.476404 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.476409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.476440 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:22.477014 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.477024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.478151 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.478157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.480016 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.480030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.481407 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.481413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.481613 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.481653 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.481760 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.481927 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":27,"thread_start_us":654,"threads_started":1}
I20260812 02:29:22.482743 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.474662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42809 (local address 127.12.8.193:44403)
0812 02:29:22.475791 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.475809 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.475989 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.476334 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.476339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.476369 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:22.476930 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.476945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.478066 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.478072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.479878 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.479892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.481635 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.481642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.481806 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.481837 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.481919 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.482041 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":53,"spinlock_wait_cycles":33024,"thread_start_us":547,"threads_started":1}
I20260812 02:29:22.483946 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6240da93c8674256b18401546b0679d6" candidate_uuid: "72504797e7f74fd4b43991a3672fb6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" is_pre_election: true
I20260812 02:29:22.483986 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6240da93c8674256b18401546b0679d6" candidate_uuid: "72504797e7f74fd4b43991a3672fb6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fd0582c11648059f535cac4b51319a" is_pre_election: true
I20260812 02:29:22.484575 15024 raft_consensus.cc:2454] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72504797e7f74fd4b43991a3672fb6e6 in term 0.
I20260812 02:29:22.484587 14862 raft_consensus.cc:2454] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72504797e7f74fd4b43991a3672fb6e6 in term 0.
I20260812 02:29:22.485406 14609 leader_election.cc:302] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fd0582c11648059f535cac4b51319a, 72504797e7f74fd4b43991a3672fb6e6; no voters:
I20260812 02:29:22.485931 15122 raft_consensus.cc:2781] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:22.486140 15122 raft_consensus.cc:482] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:22.486263 15122 raft_consensus.cc:3037] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:22.489025 15122 raft_consensus.cc:504] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.489830 15122 leader_election.cc:288] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [CANDIDATE]: Term 1 election: Requested vote from peers 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809), ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
I20260812 02:29:22.490509 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6240da93c8674256b18401546b0679d6" candidate_uuid: "72504797e7f74fd4b43991a3672fb6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6aace3e9a743c48b7e766e83bcfc02"
I20260812 02:29:22.490509 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6240da93c8674256b18401546b0679d6" candidate_uuid: "72504797e7f74fd4b43991a3672fb6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fd0582c11648059f535cac4b51319a"
I20260812 02:29:22.490782 14862 raft_consensus.cc:3037] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:22.490792 15024 raft_consensus.cc:3037] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:22.494483 14862 raft_consensus.cc:2454] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72504797e7f74fd4b43991a3672fb6e6 in term 1.
I20260812 02:29:22.494483 15024 raft_consensus.cc:2454] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72504797e7f74fd4b43991a3672fb6e6 in term 1.
I20260812 02:29:22.495123 14610 leader_election.cc:302] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72504797e7f74fd4b43991a3672fb6e6, ff6aace3e9a743c48b7e766e83bcfc02; no voters:
I20260812 02:29:22.495576 15122 raft_consensus.cc:2781] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:22.496201 15122 raft_consensus.cc:686] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 LEADER]: Becoming Leader. State: Replica: 72504797e7f74fd4b43991a3672fb6e6, State: Running, Role: LEADER
I20260812 02:29:22.496812 15122 consensus_queue.cc:237] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.502689 14499 catalog_manager.cc:5495] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193). New cstate: current_term: 1 leader_uuid: "72504797e7f74fd4b43991a3672fb6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } health_report { overall_health: UNKNOWN } } }
W20260812 02:29:22.513969 15090 tablet.cc:2369] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 02:29:22 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501760, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:22.590683 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.576402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.12.8.254:44955)
0812 02:29:22.576692 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.576697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.577462 (+ 765us) server_negotiation.cc:407] Connection header received
0812 02:29:22.577669 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.577672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.577742 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.577827 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:22.579049 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.579754 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.580952 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.581399 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.581510 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.585163 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:22.585168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:22.585171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:22.585242 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:22.588031 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:22.588720 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:22.588726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:22.588729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:22.588845 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:22.589205 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:22.589210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:22.589214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:22.589488 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:22.589651 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.590070 (+ 419us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.590378 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":109}
I20260812 02:29:22.590698 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.576415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44955 (local address 127.0.0.1:38788)
0812 02:29:22.577336 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.577356 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.577548 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.577878 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.577883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.578606 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:22.578907 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.578917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.579885 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.579890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.580820 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.580832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.581589 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.581596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.581746 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.582493 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:22.582511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:22.584991 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:22.588167 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:22.588180 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:22.588186 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:22.588610 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:22.588943 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:22.588947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:22.588951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:22.589075 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:22.589615 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:22.589624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:22.589801 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.589948 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.590230 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":472,"threads_started":1}
I20260812 02:29:22.597776 14499 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:38788:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:29:22.599362 14499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:22.608628 15003 tablet_service.cc:1467] Processing CreateTablet for tablet c14c2956edfd48cbb5b2dc8486ebf71c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37b0ab98fa45441da2caca0588844cb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:29:22.608428 14671 tablet_service.cc:1467] Processing CreateTablet for tablet c14c2956edfd48cbb5b2dc8486ebf71c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37b0ab98fa45441da2caca0588844cb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:29:22.608558 14840 tablet_service.cc:1467] Processing CreateTablet for tablet c14c2956edfd48cbb5b2dc8486ebf71c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37b0ab98fa45441da2caca0588844cb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:29:22.609735 15003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14c2956edfd48cbb5b2dc8486ebf71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:22.609815 14840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14c2956edfd48cbb5b2dc8486ebf71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:22.609817 14671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14c2956edfd48cbb5b2dc8486ebf71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:22.617055 15112 tablet_bootstrap.cc:492] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: Bootstrap starting.
I20260812 02:29:22.618461 15115 tablet_bootstrap.cc:492] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: Bootstrap starting.
I20260812 02:29:22.618829 15112 tablet_bootstrap.cc:654] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:22.619199 15114 tablet_bootstrap.cc:492] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: Bootstrap starting.
I20260812 02:29:22.620630 15115 tablet_bootstrap.cc:654] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:22.621100 15114 tablet_bootstrap.cc:654] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:22.621117 15112 tablet_bootstrap.cc:492] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: No bootstrap required, opened a new log
I20260812 02:29:22.621299 15112 ts_tablet_manager.cc:1397] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:22.622072 15112 raft_consensus.cc:348] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.622346 15112 raft_consensus.cc:374] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:22.622435 15112 raft_consensus.cc:729] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6aace3e9a743c48b7e766e83bcfc02, State: Initialized, Role: FOLLOWER
I20260812 02:29:22.623467 15112 consensus_queue.cc:260] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.624256 15115 tablet_bootstrap.cc:492] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: No bootstrap required, opened a new log
I20260812 02:29:22.624009 15119 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.624395 15119 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:22.624444 15115 ts_tablet_manager.cc:1397] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260812 02:29:22.624884 15114 tablet_bootstrap.cc:492] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: No bootstrap required, opened a new log
I20260812 02:29:22.625059 15114 ts_tablet_manager.cc:1397] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:29:22.625254 15112 ts_tablet_manager.cc:1428] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:29:22.625420 15115 raft_consensus.cc:348] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.625746 15115 raft_consensus.cc:374] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:22.625867 15115 raft_consensus.cc:729] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fd0582c11648059f535cac4b51319a, State: Initialized, Role: FOLLOWER
I20260812 02:29:22.626003 15114 raft_consensus.cc:348] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.626773 15114 raft_consensus.cc:374] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:22.626536 15115 consensus_queue.cc:260] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.627624 15114 raft_consensus.cc:729] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72504797e7f74fd4b43991a3672fb6e6, State: Initialized, Role: FOLLOWER
I20260812 02:29:22.627947 15114 consensus_queue.cc:260] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.628196 15121 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.628662 15121 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:22.628700 15114 ts_tablet_manager.cc:1428] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 02:29:22.629213 15115 ts_tablet_manager.cc:1428] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 02:29:22.629127 15122 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.629719 15122 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:22.739005 15121 raft_consensus.cc:482] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:22.739347 15121 raft_consensus.cc:504] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.741233 15121 leader_election.cc:288] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887), ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
I20260812 02:29:22.749452 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.741720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:41631 (local address 127.12.8.193:32887)
0812 02:29:22.741967 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.741973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.742975 (+ 1002us) server_negotiation.cc:407] Connection header received
0812 02:29:22.743251 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.743257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.743349 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.743447 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:22.744363 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.745135 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.747473 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.748277 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.748411 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.748472 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.748983 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.749126 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 02:29:22.749647 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.741727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:32887 (local address 127.12.8.194:41631)
0812 02:29:22.742870 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.742891 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.743053 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.743509 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.743515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.743546 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:22.744130 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.744144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.745308 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.745315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.747288 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.747304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.748532 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.748541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.748734 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.748782 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.748899 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.749044 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":5,"spinlock_wait_cycles":11136,"thread_start_us":949,"threads_started":1}
I20260812 02:29:22.749907 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.741776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:41797 (local address 127.12.8.194:35721)
0812 02:29:22.742932 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.742955 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.743156 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.743784 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.743790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.743822 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:22.744333 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.744348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.745658 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.745662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.747355 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.747368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.748852 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.748857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.749052 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.749093 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.749274 (+ 181us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.749427 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":82,"thread_start_us":527,"threads_started":1}
I20260812 02:29:22.750239 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.741848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:35721 (local address 127.12.8.195:41797)
0812 02:29:22.742085 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.742093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.743095 (+ 1002us) server_negotiation.cc:407] Connection header received
0812 02:29:22.743402 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.743409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.743494 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.743582 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:22.744540 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.745510 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.747553 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.748455 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.748654 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.748751 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.749314 (+ 563us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.749492 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 02:29:22.750914 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14c2956edfd48cbb5b2dc8486ebf71c" candidate_uuid: "25fd0582c11648059f535cac4b51319a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" is_pre_election: true
I20260812 02:29:22.751194 15024 raft_consensus.cc:2454] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25fd0582c11648059f535cac4b51319a in term 0.
I20260812 02:29:22.751288 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14c2956edfd48cbb5b2dc8486ebf71c" candidate_uuid: "25fd0582c11648059f535cac4b51319a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72504797e7f74fd4b43991a3672fb6e6" is_pre_election: true
I20260812 02:29:22.751796 14779 leader_election.cc:302] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fd0582c11648059f535cac4b51319a, ff6aace3e9a743c48b7e766e83bcfc02; no voters:
I20260812 02:29:22.751883 14694 raft_consensus.cc:2454] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25fd0582c11648059f535cac4b51319a in term 0.
I20260812 02:29:22.752264 15121 raft_consensus.cc:2781] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:22.752451 15121 raft_consensus.cc:482] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:22.752601 15121 raft_consensus.cc:3037] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:22.755590 15121 raft_consensus.cc:504] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.756436 15121 leader_election.cc:288] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [CANDIDATE]: Term 1 election: Requested vote from peers 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887), ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
I20260812 02:29:22.757285 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14c2956edfd48cbb5b2dc8486ebf71c" candidate_uuid: "25fd0582c11648059f535cac4b51319a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72504797e7f74fd4b43991a3672fb6e6"
I20260812 02:29:22.757258 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14c2956edfd48cbb5b2dc8486ebf71c" candidate_uuid: "25fd0582c11648059f535cac4b51319a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6aace3e9a743c48b7e766e83bcfc02"
I20260812 02:29:22.757560 14694 raft_consensus.cc:3037] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:22.757597 15024 raft_consensus.cc:3037] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:22.761154 15024 raft_consensus.cc:2454] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25fd0582c11648059f535cac4b51319a in term 1.
I20260812 02:29:22.761236 14694 raft_consensus.cc:2454] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25fd0582c11648059f535cac4b51319a in term 1.
I20260812 02:29:22.761830 14779 leader_election.cc:302] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25fd0582c11648059f535cac4b51319a, 72504797e7f74fd4b43991a3672fb6e6, ff6aace3e9a743c48b7e766e83bcfc02; no voters:
I20260812 02:29:22.762333 15121 raft_consensus.cc:2781] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:22.762950 15121 raft_consensus.cc:686] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [term 1 LEADER]: Becoming Leader. State: Replica: 25fd0582c11648059f535cac4b51319a, State: Running, Role: LEADER
I20260812 02:29:22.763567 15121 consensus_queue.cc:237] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:22.767659 15142 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: TxnStatusTablet state changed. Reason: New leader 25fd0582c11648059f535cac4b51319a. Latest consensus state: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.768172 15142 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:29:22.769302 14500 catalog_manager.cc:5495] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a reported cstate change: term changed from 0 to 1, leader changed from <none> to 25fd0582c11648059f535cac4b51319a (127.12.8.194). New cstate: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } health_report { overall_health: UNKNOWN } } }
I20260812 02:29:22.770859 15144 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:29:22.771205 15144 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:29:22.791301 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.784521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42809 (local address 127.0.0.1:33548)
0812 02:29:22.784919 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.784937 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.785136 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.785784 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.785789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.785803 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:29:22.786027 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.786050 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.787307 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.787313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.788567 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.788581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.789383 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.789389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.789594 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.789667 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:29:22.790674 (+ 1007us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:29:22.790699 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.790787 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.790962 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260812 02:29:22.791328 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.784500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.12.8.194:42809)
0812 02:29:22.784706 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.784711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.785035 (+ 324us) server_negotiation.cc:407] Connection header received
0812 02:29:22.785438 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.785447 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.785559 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.785745 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:22.786193 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.787178 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.788710 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.789216 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.789350 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.789870 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:22.790560 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:22.790715 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.790808 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.790900 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:29:22.801503 14694 raft_consensus.cc:1261] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Refusing update from remote peer 25fd0582c11648059f535cac4b51319a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:22.801604 15024 raft_consensus.cc:1261] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Refusing update from remote peer 25fd0582c11648059f535cac4b51319a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:22.802613 15121 consensus_queue.cc:1035] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:22.803061 15142 consensus_queue.cc:1035] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:22.817274 15122 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: TxnStatusTablet state changed. Reason: New leader 25fd0582c11648059f535cac4b51319a. Latest consensus state: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.817719 15122 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:22.819576 15119 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: TxnStatusTablet state changed. Reason: New leader 25fd0582c11648059f535cac4b51319a. Latest consensus state: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.819905 15119 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:22.823956 15147 mvcc.cc:205] Tried to move back new op lower bound from 7317511220422529024 to 7317511220283662336. Current Snapshot: MvccSnapshot[applied={T|T < 7317511220422529024}]
I20260812 02:29:22.825090 15122 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.825475 15122 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P 72504797e7f74fd4b43991a3672fb6e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:22.829903 15121 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.830338 15121 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:29:22.839572 15149 mvcc.cc:205] Tried to move back new op lower bound from 7317511220422529024 to 7317511220283662336. Current Snapshot: MvccSnapshot[applied={T|T < 7317511220422529024}]
I20260812 02:29:22.840585 15119 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.840891 15119 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P ff6aace3e9a743c48b7e766e83bcfc02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:29:22.846685 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.838381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.12.8.193:32887)
0812 02:29:22.838570 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.838576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.838599 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:29:22.838911 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.838918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.839077 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.839239 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:22.840050 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.841231 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.843283 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.843926 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.844033 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.844897 (+ 864us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:22.845851 (+ 954us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:22.845978 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.846264 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.846377 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:29:22.846997 15152 tablet_replica.cc:439] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } } }
I20260812 02:29:22.847512 15152 tablet_replica.cc:442] T c14c2956edfd48cbb5b2dc8486ebf71c P 25fd0582c11648059f535cac4b51319a: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:29:22.871265 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.863754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:33519 (local address 127.12.8.194:42809)
0812 02:29:22.864046 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.864053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.864643 (+ 590us) server_negotiation.cc:407] Connection header received
0812 02:29:22.864859 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.864866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.864970 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.865106 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:22.865777 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.866954 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.868958 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.869683 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.869840 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.870447 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:22.870606 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:22.870701 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.870843 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.870940 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 02:29:22.871316 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.863391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42809 (local address 127.12.8.193:33519)
0812 02:29:22.864499 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.864520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.864706 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.865155 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.865159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.865201 (+ 42us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:29:22.865593 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.865604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.867220 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.867228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.868768 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.868788 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.869911 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.869917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.870097 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.870195 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:29:22.870710 (+ 515us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:29:22.870718 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.870779 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.870939 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":681,"threads_started":1}
I20260812 02:29:22.885231 15024 raft_consensus.cc:1261] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Refusing update from remote peer 72504797e7f74fd4b43991a3672fb6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:22.885511 14862 raft_consensus.cc:1261] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Refusing update from remote peer 72504797e7f74fd4b43991a3672fb6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:22.885993 15122 consensus_queue.cc:1035] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:22.886751 15122 consensus_queue.cc:1035] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:22.936540 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.928027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:53981 (local address 127.12.8.193:32887)
0812 02:29:22.928302 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:22.928309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:22.928880 (+ 571us) server_negotiation.cc:407] Connection header received
0812 02:29:22.929250 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:22.929257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:22.929370 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:22.929464 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:22.930117 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.931084 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.933683 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.934386 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.934528 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.935340 (+ 812us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:22.935557 (+ 217us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:22.935685 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:22.936078 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 02:29:22.936252 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 02:29:22.937479 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:22.927321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:32887 (local address 127.12.8.194:53981)
0812 02:29:22.928776 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:22.928797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:22.928961 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:22.929512 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:22.929517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:22.929530 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:29:22.929926 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.929941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.931240 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.931245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.933446 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:22.933463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:22.934696 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:22.934706 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:22.934974 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:22.935076 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:29:22.935818 (+ 742us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:29:22.935832 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 02:29:22.936848 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0812 02:29:22.937016 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"thread_start_us":494,"threads_started":1}
I20260812 02:29:22.993489 14500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:22.993908 14500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:22.999094 14500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: Sending DeleteTablet for 3 replicas of tablet 6240da93c8674256b18401546b0679d6
I20260812 02:29:23.000993 14671 tablet_service.cc:1514] Processing DeleteTablet for tablet 6240da93c8674256b18401546b0679d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:22 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51460
I20260812 02:29:23.001916 15003 tablet_service.cc:1514] Processing DeleteTablet for tablet 6240da93c8674256b18401546b0679d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:22 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:46316
I20260812 02:29:23.002820 14840 tablet_service.cc:1514] Processing DeleteTablet for tablet 6240da93c8674256b18401546b0679d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:22 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33540
I20260812 02:29:23.002825 14500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:23.003664 14500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:23.006569 15174 tablet_replica.cc:330] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: stopping tablet replica
I20260812 02:29:23.007256 15174 raft_consensus.cc:2229] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:23.007716 15174 raft_consensus.cc:2258] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:23.010396 15174 ts_tablet_manager.cc:1905] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:23.016788 14669 tablet_service.cc:1467] Processing CreateTablet for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 (DEFAULT_TABLE table=test-table [id=6910da92b74a447da9698e00cbee5671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:23.016906 14839 tablet_service.cc:1467] Processing CreateTablet for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 (DEFAULT_TABLE table=test-table [id=6910da92b74a447da9698e00cbee5671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:23.017875 14669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:23.017875 14839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:23.023305 15003 tablet_service.cc:1467] Processing CreateTablet for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 (DEFAULT_TABLE table=test-table [id=6910da92b74a447da9698e00cbee5671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:23.032883 15114 tablet_bootstrap.cc:492] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: Bootstrap starting.
I20260812 02:29:23.033317 15003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:23.035800 15114 tablet_bootstrap.cc:654] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:23.036640 15175 tablet_replica.cc:330] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: stopping tablet replica
I20260812 02:29:23.037668 15175 raft_consensus.cc:2229] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:23.038112 15172 tablet_replica.cc:330] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: stopping tablet replica
I20260812 02:29:23.038586 15172 raft_consensus.cc:2229] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:29:23.039105 15175 raft_consensus.cc:2258] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:23.041895 15175 ts_tablet_manager.cc:1905] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:23.042264 15172 raft_consensus.cc:2258] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:23.044129 15172 ts_tablet_manager.cc:1905] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:23.052889 15115 tablet_bootstrap.cc:492] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: Bootstrap starting.
I20260812 02:29:23.055814 15115 tablet_bootstrap.cc:654] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:23.057862 15172 ts_tablet_manager.cc:1918] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:29:23.058145 15172 log.cc:1196] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/6240da93c8674256b18401546b0679d6
I20260812 02:29:23.058300 15175 ts_tablet_manager.cc:1918] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:29:23.058470 15175 log.cc:1196] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/6240da93c8674256b18401546b0679d6
I20260812 02:29:23.058988 15175 ts_tablet_manager.cc:1939] T 6240da93c8674256b18401546b0679d6 P 25fd0582c11648059f535cac4b51319a: Deleting consensus metadata
I20260812 02:29:23.059265 15172 ts_tablet_manager.cc:1939] T 6240da93c8674256b18401546b0679d6 P 72504797e7f74fd4b43991a3672fb6e6: Deleting consensus metadata
I20260812 02:29:23.060936 14486 catalog_manager.cc:4841] TS 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809): tablet 6240da93c8674256b18401546b0679d6 (table test-table [id=0d792fb48a364f2da3805faa1ee4ef7b]) successfully deleted
I20260812 02:29:23.061093 15112 tablet_bootstrap.cc:492] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: Bootstrap starting.
I20260812 02:29:23.061560 14486 catalog_manager.cc:4841] TS 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887): tablet 6240da93c8674256b18401546b0679d6 (table test-table [id=0d792fb48a364f2da3805faa1ee4ef7b]) successfully deleted
I20260812 02:29:23.062897 15115 tablet_bootstrap.cc:492] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: No bootstrap required, opened a new log
I20260812 02:29:23.062932 15112 tablet_bootstrap.cc:654] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:23.063163 15115 ts_tablet_manager.cc:1397] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260812 02:29:23.064215 15115 raft_consensus.cc:348] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.064623 15115 raft_consensus.cc:374] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:23.064743 15115 raft_consensus.cc:729] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fd0582c11648059f535cac4b51319a, State: Initialized, Role: FOLLOWER
I20260812 02:29:23.065104 15115 consensus_queue.cc:260] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.068710 15115 ts_tablet_manager.cc:1428] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260812 02:29:23.068997 15174 ts_tablet_manager.cc:1918] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:29:23.069203 15174 log.cc:1196] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/6240da93c8674256b18401546b0679d6
I20260812 02:29:23.069756 15174 ts_tablet_manager.cc:1939] T 6240da93c8674256b18401546b0679d6 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting consensus metadata
I20260812 02:29:23.071729 14487 catalog_manager.cc:4841] TS ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797): tablet 6240da93c8674256b18401546b0679d6 (table test-table [id=0d792fb48a364f2da3805faa1ee4ef7b]) successfully deleted
I20260812 02:29:23.074864 15114 tablet_bootstrap.cc:492] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: No bootstrap required, opened a new log
I20260812 02:29:23.075160 15114 ts_tablet_manager.cc:1397] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: Time spent bootstrapping tablet: real 0.042s user 0.021s sys 0.000s
I20260812 02:29:23.076131 15114 raft_consensus.cc:348] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.076526 15114 raft_consensus.cc:374] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:23.076790 15114 raft_consensus.cc:729] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72504797e7f74fd4b43991a3672fb6e6, State: Initialized, Role: FOLLOWER
I20260812 02:29:23.077355 15112 tablet_bootstrap.cc:492] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: No bootstrap required, opened a new log
I20260812 02:29:23.077207 15114 consensus_queue.cc:260] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.077522 15112 ts_tablet_manager.cc:1397] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260812 02:29:23.078037 15114 ts_tablet_manager.cc:1428] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 02:29:23.078305 15112 raft_consensus.cc:348] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.078513 15112 raft_consensus.cc:374] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:23.078589 15112 raft_consensus.cc:729] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6aace3e9a743c48b7e766e83bcfc02, State: Initialized, Role: FOLLOWER
I20260812 02:29:23.078824 15112 consensus_queue.cc:260] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.081327 15119 raft_consensus.cc:482] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:23.081511 15119 raft_consensus.cc:504] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.083317 15119 leader_election.cc:288] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887), 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809)
I20260812 02:29:23.089625 15112 ts_tablet_manager.cc:1428] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260812 02:29:23.093698 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.083924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:38301 (local address 127.12.8.193:32887)
0812 02:29:23.084171 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:23.084179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:23.084686 (+ 507us) server_negotiation.cc:407] Connection header received
0812 02:29:23.084975 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:23.084981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:23.085059 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:23.085122 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:23.086010 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.087429 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.090322 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.091668 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.091816 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.091893 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:23.093142 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0812 02:29:23.093328 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:29:23.094825 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.083873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:32887 (local address 127.12.8.195:38301)
0812 02:29:23.084580 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:23.084599 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:23.084804 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:23.085386 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:23.085390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:23.085417 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:23.085784 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:23.085796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.087731 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.087737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:23.090127 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:23.090141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.092074 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.092082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:23.092939 (+ 857us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.092976 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 02:29:23.094181 (+ 1205us) client_negotiation.cc:240] Negotiation successful
0812 02:29:23.094364 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":389,"threads_started":1}
I20260812 02:29:23.098963 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bbb4f6cbc04f4e9d502d4292cf60b5" candidate_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72504797e7f74fd4b43991a3672fb6e6" is_pre_election: true
I20260812 02:29:23.099260 14694 raft_consensus.cc:2454] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6aace3e9a743c48b7e766e83bcfc02 in term 0.
I20260812 02:29:23.099867 14947 leader_election.cc:302] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72504797e7f74fd4b43991a3672fb6e6, ff6aace3e9a743c48b7e766e83bcfc02; no voters:
I20260812 02:29:23.100301 15119 raft_consensus.cc:2781] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:23.100423 15119 raft_consensus.cc:482] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:23.100505 15119 raft_consensus.cc:3037] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:23.102938 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.085126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:43351 (local address 127.12.8.194:42809)
0812 02:29:23.085404 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:23.085409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:23.095380 (+ 9971us) server_negotiation.cc:407] Connection header received
0812 02:29:23.095683 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:23.095689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:23.095787 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:23.095850 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:23.096440 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.097767 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.100166 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.101546 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.101691 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.101787 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:23.102494 (+ 707us) server_negotiation.cc:299] Negotiation successful
0812 02:29:23.102646 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260812 02:29:23.103806 15119 raft_consensus.cc:504] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.104712 15119 leader_election.cc:288] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [CANDIDATE]: Term 1 election: Requested vote from peers 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887), 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809)
I20260812 02:29:23.103798 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.084658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42809 (local address 127.12.8.195:43351)
0812 02:29:23.095243 (+ 10585us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:23.095264 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:23.095513 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:23.096063 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:23.096068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:23.096093 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:23.096273 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:23.096283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.097951 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.097956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:23.099893 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:23.099908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.102057 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.102066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:23.102266 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.102335 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 02:29:23.103339 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0812 02:29:23.103490 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10471,"thread_start_us":13519,"threads_started":1}
I20260812 02:29:23.105306 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bbb4f6cbc04f4e9d502d4292cf60b5" candidate_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72504797e7f74fd4b43991a3672fb6e6"
I20260812 02:29:23.105541 14694 raft_consensus.cc:3037] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:23.108320 14694 raft_consensus.cc:2454] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6aace3e9a743c48b7e766e83bcfc02 in term 1.
I20260812 02:29:23.109403 14947 leader_election.cc:302] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72504797e7f74fd4b43991a3672fb6e6, ff6aace3e9a743c48b7e766e83bcfc02; no voters:
I20260812 02:29:23.109684 15119 raft_consensus.cc:2781] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:23.110200 15119 raft_consensus.cc:686] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 LEADER]: Becoming Leader. State: Replica: ff6aace3e9a743c48b7e766e83bcfc02, State: Running, Role: LEADER
I20260812 02:29:23.110806 15119 consensus_queue.cc:237] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:23.112898 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bbb4f6cbc04f4e9d502d4292cf60b5" candidate_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fd0582c11648059f535cac4b51319a" is_pre_election: true
I20260812 02:29:23.113216 14862 raft_consensus.cc:2454] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6aace3e9a743c48b7e766e83bcfc02 in term 0.
I20260812 02:29:23.113957 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bbb4f6cbc04f4e9d502d4292cf60b5" candidate_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fd0582c11648059f535cac4b51319a"
I20260812 02:29:23.114140 14862 raft_consensus.cc:3037] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:23.116606 14862 raft_consensus.cc:2454] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6aace3e9a743c48b7e766e83bcfc02 in term 1.
I20260812 02:29:23.119266 14503 catalog_manager.cc:5495] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195). New cstate: current_term: 1 leader_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } health_report { overall_health: HEALTHY } } }
I20260812 02:29:23.145485 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.139738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.12.8.195:41797)
0812 02:29:23.139995 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:23.140002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:23.140025 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:29:23.140159 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:23.140164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:23.140288 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:23.140593 (+ 305us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:23.141070 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.141968 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.143282 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.143691 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.143799 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.144227 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:23.144827 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:23.144980 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:23.145066 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 02:29:23.145156 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:29:23.161612 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.156053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:52173 (local address 127.12.8.194:42809)
0812 02:29:23.156256 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:23.156262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:23.156478 (+ 216us) server_negotiation.cc:407] Connection header received
0812 02:29:23.156736 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:23.156741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:23.156820 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:23.156894 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:23.157394 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.158266 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.159776 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.160264 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.160379 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.160830 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:23.160945 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:23.161024 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:23.161152 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 02:29:23.161249 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:29:23.161666 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.155726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42809 (local address 127.12.8.195:52173)
0812 02:29:23.156377 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:23.156397 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:23.156596 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:23.156925 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:23.156929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:23.156941 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:29:23.157241 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:23.157253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.158413 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.158419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:23.159634 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:23.159647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.160441 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.160448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:23.160570 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.160626 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:29:23.161045 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:29:23.161056 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 02:29:23.161123 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 02:29:23.161310 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":543,"threads_started":1}
I20260812 02:29:23.172683 14862 raft_consensus.cc:1261] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Refusing update from remote peer ff6aace3e9a743c48b7e766e83bcfc02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:23.172649 14694 raft_consensus.cc:1261] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Refusing update from remote peer ff6aace3e9a743c48b7e766e83bcfc02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:23.173489 15182 consensus_queue.cc:1035] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:23.173974 15119 consensus_queue.cc:1035] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:23.243575 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.237210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:41797 (local address 127.12.8.194:43285)
0812 02:29:23.237447 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:23.237473 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:23.237600 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:23.238657 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:23.238662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:23.238676 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:29:23.238908 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:23.238920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.240088 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.240094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:23.241405 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:23.241421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.242235 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.242241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:23.242458 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.242515 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:29:23.242982 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:29:23.242994 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 02:29:23.243075 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 02:29:23.243223 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 02:29:23.244234 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:23.238007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:43285 (local address 127.12.8.195:41797)
0812 02:29:23.238199 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:23.238205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:23.238226 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:29:23.238294 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:23.238298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:23.238394 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:23.238480 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:23.239080 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.239870 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.241584 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:23.242074 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:23.242183 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:23.242708 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:23.242857 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:23.242935 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:23.243836 (+ 901us) server_negotiation.cc:299] Negotiation successful
0812 02:29:23.243920 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 02:29:23.245718 15149 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 02:29:24.896211 14503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:24.896545 14503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:24.903863 14503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: Sending DeleteTablet for 3 replicas of tablet 64bbb4f6cbc04f4e9d502d4292cf60b5
I20260812 02:29:24.905040 14671 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:24 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51460
I20260812 02:29:24.905431 14839 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:24 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33540
I20260812 02:29:24.905396 15003 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:24 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:46316
I20260812 02:29:24.906473 15225 tablet_replica.cc:330] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: stopping tablet replica
I20260812 02:29:24.906785 15225 raft_consensus.cc:2229] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:24.907310 15225 raft_consensus.cc:2258] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:24.907534 15227 tablet_replica.cc:330] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: stopping tablet replica
I20260812 02:29:24.907823 15227 raft_consensus.cc:2229] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:29:24.908484 15227 raft_consensus.cc:2258] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:24.908710 15225 ts_tablet_manager.cc:1905] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:24.909662 15227 ts_tablet_manager.cc:1905] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:24.918745 15225 ts_tablet_manager.cc:1918] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:29:24.918977 15227 ts_tablet_manager.cc:1918] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:29:24.919196 15227 log.cc:1196] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/64bbb4f6cbc04f4e9d502d4292cf60b5
I20260812 02:29:24.919207 15225 log.cc:1196] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/64bbb4f6cbc04f4e9d502d4292cf60b5
I20260812 02:29:24.919876 15227 ts_tablet_manager.cc:1939] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting consensus metadata
I20260812 02:29:24.920075 15225 ts_tablet_manager.cc:1939] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 72504797e7f74fd4b43991a3672fb6e6: Deleting consensus metadata
I20260812 02:29:24.921336 14487 catalog_manager.cc:4841] TS ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797): tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 (table test-table [id=6910da92b74a447da9698e00cbee5671]) successfully deleted
I20260812 02:29:24.921470 14486 catalog_manager.cc:4841] TS 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887): tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 (table test-table [id=6910da92b74a447da9698e00cbee5671]) successfully deleted
I20260812 02:29:24.923964 15226 tablet_replica.cc:330] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: stopping tablet replica
I20260812 02:29:24.924350 15226 raft_consensus.cc:2229] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:24.926375 15226 raft_consensus.cc:2258] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:24.927951 15226 ts_tablet_manager.cc:1905] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:24.936017 15226 ts_tablet_manager.cc:1918] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:29:24.936343 15226 log.cc:1196] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/64bbb4f6cbc04f4e9d502d4292cf60b5
I20260812 02:29:24.937151 15226 ts_tablet_manager.cc:1939] T 64bbb4f6cbc04f4e9d502d4292cf60b5 P 25fd0582c11648059f535cac4b51319a: Deleting consensus metadata
I20260812 02:29:24.938781 14486 catalog_manager.cc:4841] TS 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809): tablet 64bbb4f6cbc04f4e9d502d4292cf60b5 (table test-table [id=6910da92b74a447da9698e00cbee5671]) successfully deleted
Aug 12 02:29:24 dist-test-slave-vq2q krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501764, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:29:24 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501764, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:24.994411 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:24.967994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.12.8.254:44955)
0812 02:29:24.969343 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:24.969351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:24.969415 (+ 64us) server_negotiation.cc:407] Connection header received
0812 02:29:24.969555 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:24.969560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:24.969651 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:24.969777 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:24.972040 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:24.973426 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:24.975330 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:24.976030 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:24.976202 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:24.986153 (+ 9951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:24.986163 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:24.986167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:24.986229 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:24.990896 (+ 4667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:24.991769 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:24.991773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:24.991776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:24.991971 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:24.992411 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:24.992414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:24.992417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:24.992885 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:24.993265 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:24.993531 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 02:29:24.993790 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":668,"threads_started":1}
I20260812 02:29:24.999579 14503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:25.000540 14503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:25.009537 14671 tablet_service.cc:1467] Processing CreateTablet for tablet c9544c66e0964254b2b4de5acbb850d6 (DEFAULT_TABLE table=test-table [id=aabf89ded5bb47ef9683843f8146781e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:25.009613 14839 tablet_service.cc:1467] Processing CreateTablet for tablet c9544c66e0964254b2b4de5acbb850d6 (DEFAULT_TABLE table=test-table [id=aabf89ded5bb47ef9683843f8146781e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:25.009696 15003 tablet_service.cc:1467] Processing CreateTablet for tablet c9544c66e0964254b2b4de5acbb850d6 (DEFAULT_TABLE table=test-table [id=aabf89ded5bb47ef9683843f8146781e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:25.010109 15003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9544c66e0964254b2b4de5acbb850d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:25.010187 14671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9544c66e0964254b2b4de5acbb850d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:25.010219 14839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9544c66e0964254b2b4de5acbb850d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:25.017587 15236 tablet_bootstrap.cc:492] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: Bootstrap starting.
I20260812 02:29:25.018770 15238 tablet_bootstrap.cc:492] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: Bootstrap starting.
I20260812 02:29:25.020834 15236 tablet_bootstrap.cc:654] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:25.021584 15238 tablet_bootstrap.cc:654] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:25.024343 15236 tablet_bootstrap.cc:492] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: No bootstrap required, opened a new log
I20260812 02:29:25.024576 15236 ts_tablet_manager.cc:1397] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260812 02:29:25.025803 15236 raft_consensus.cc:348] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.026134 15236 raft_consensus.cc:374] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:25.026293 15236 raft_consensus.cc:729] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6aace3e9a743c48b7e766e83bcfc02, State: Initialized, Role: FOLLOWER
I20260812 02:29:25.026628 15236 consensus_queue.cc:260] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.027421 15236 ts_tablet_manager.cc:1428] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:29:25.027925 15240 tablet_bootstrap.cc:492] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: Bootstrap starting.
I20260812 02:29:25.030624 15238 tablet_bootstrap.cc:492] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: No bootstrap required, opened a new log
I20260812 02:29:25.030869 15238 ts_tablet_manager.cc:1397] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260812 02:29:25.031639 15240 tablet_bootstrap.cc:654] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:25.032691 15238 raft_consensus.cc:348] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.033109 15238 raft_consensus.cc:374] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:25.033350 15238 raft_consensus.cc:729] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fd0582c11648059f535cac4b51319a, State: Initialized, Role: FOLLOWER
I20260812 02:29:25.033679 15238 consensus_queue.cc:260] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.034533 15238 ts_tablet_manager.cc:1428] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260812 02:29:25.035811 15240 tablet_bootstrap.cc:492] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: No bootstrap required, opened a new log
I20260812 02:29:25.036087 15240 ts_tablet_manager.cc:1397] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260812 02:29:25.037545 15240 raft_consensus.cc:348] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.037971 15240 raft_consensus.cc:374] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:25.038221 15240 raft_consensus.cc:729] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72504797e7f74fd4b43991a3672fb6e6, State: Initialized, Role: FOLLOWER
I20260812 02:29:25.038611 15240 consensus_queue.cc:260] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.040377 15240 ts_tablet_manager.cc:1428] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260812 02:29:25.107522 15194 raft_consensus.cc:482] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:25.107946 15194 raft_consensus.cc:504] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.109047 15194 leader_election.cc:288] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887), ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
I20260812 02:29:25.110057 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9544c66e0964254b2b4de5acbb850d6" candidate_uuid: "25fd0582c11648059f535cac4b51319a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" is_pre_election: true
I20260812 02:29:25.110035 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9544c66e0964254b2b4de5acbb850d6" candidate_uuid: "25fd0582c11648059f535cac4b51319a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72504797e7f74fd4b43991a3672fb6e6" is_pre_election: true
I20260812 02:29:25.110481 15024 raft_consensus.cc:2454] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25fd0582c11648059f535cac4b51319a in term 0.
I20260812 02:29:25.110562 14694 raft_consensus.cc:2454] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25fd0582c11648059f535cac4b51319a in term 0.
I20260812 02:29:25.111294 14779 leader_election.cc:302] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25fd0582c11648059f535cac4b51319a, 72504797e7f74fd4b43991a3672fb6e6, ff6aace3e9a743c48b7e766e83bcfc02; no voters:
I20260812 02:29:25.111835 15194 raft_consensus.cc:2781] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:25.112093 15194 raft_consensus.cc:482] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:25.112260 15194 raft_consensus.cc:3037] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:25.115387 15194 raft_consensus.cc:504] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.116168 15194 leader_election.cc:288] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [CANDIDATE]: Term 1 election: Requested vote from peers 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887), ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
I20260812 02:29:25.117067 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9544c66e0964254b2b4de5acbb850d6" candidate_uuid: "25fd0582c11648059f535cac4b51319a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6aace3e9a743c48b7e766e83bcfc02"
I20260812 02:29:25.117194 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9544c66e0964254b2b4de5acbb850d6" candidate_uuid: "25fd0582c11648059f535cac4b51319a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72504797e7f74fd4b43991a3672fb6e6"
I20260812 02:29:25.117496 15024 raft_consensus.cc:3037] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:25.117602 14694 raft_consensus.cc:3037] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:25.120980 15024 raft_consensus.cc:2454] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25fd0582c11648059f535cac4b51319a in term 1.
I20260812 02:29:25.121268 14694 raft_consensus.cc:2454] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25fd0582c11648059f535cac4b51319a in term 1.
I20260812 02:29:25.121582 14779 leader_election.cc:302] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fd0582c11648059f535cac4b51319a, ff6aace3e9a743c48b7e766e83bcfc02; no voters:
I20260812 02:29:25.122104 15194 raft_consensus.cc:2781] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:25.122398 15194 raft_consensus.cc:686] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 1 LEADER]: Becoming Leader. State: Replica: 25fd0582c11648059f535cac4b51319a, State: Running, Role: LEADER
I20260812 02:29:25.122706 15194 consensus_queue.cc:237] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.126924 14503 catalog_manager.cc:5495] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a reported cstate change: term changed from 0 to 1, leader changed from <none> to 25fd0582c11648059f535cac4b51319a (127.12.8.194). New cstate: current_term: 1 leader_uuid: "25fd0582c11648059f535cac4b51319a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } health_report { overall_health: UNKNOWN } } }
I20260812 02:29:25.168169 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:25.159772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.12.8.194:42809)
0812 02:29:25.161087 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:25.161092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:25.161139 (+ 47us) server_negotiation.cc:407] Connection header received
0812 02:29:25.161270 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:25.161275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:25.161356 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:25.161465 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:25.162231 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:25.163644 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:25.165507 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:25.166252 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:25.166392 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:25.167134 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:25.167281 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:25.167364 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:25.167546 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 02:29:25.167643 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":823,"threads_started":1}
I20260812 02:29:25.189811 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:25.180977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:39345 (local address 127.12.8.194:42809)
0812 02:29:25.181360 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:25.181365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:25.182000 (+ 635us) server_negotiation.cc:407] Connection header received
0812 02:29:25.182208 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:25.182215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:25.182439 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:25.182619 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:25.183400 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:25.184485 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:25.186932 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:25.188032 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:25.188191 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:25.188795 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:25.188963 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:25.189066 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:25.189263 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 02:29:25.189377 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:29:25.190024 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:25.180574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42809 (local address 127.12.8.194:39345)
0812 02:29:25.181887 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:25.181906 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:25.182064 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:25.182640 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:25.182644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:25.182654 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:29:25.183232 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:25.183245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:25.184728 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:25.184732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:25.186745 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:25.186762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:25.188319 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:25.188325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:25.188558 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:25.188607 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:29:25.189085 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:29:25.189101 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 02:29:25.189203 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 02:29:25.189335 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"thread_start_us":817,"threads_started":1}
I20260812 02:29:25.201258 14694 raft_consensus.cc:1261] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Refusing update from remote peer 25fd0582c11648059f535cac4b51319a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:25.201874 15024 raft_consensus.cc:1261] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Refusing update from remote peer 25fd0582c11648059f535cac4b51319a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:25.202061 15217 consensus_queue.cc:1035] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:25.202652 15194 consensus_queue.cc:1035] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:25.272790 14503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:25.273340 14503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:25.278242 14503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: Sending DeleteTablet for 3 replicas of tablet c9544c66e0964254b2b4de5acbb850d6
I20260812 02:29:25.279707 14671 tablet_service.cc:1514] Processing DeleteTablet for tablet c9544c66e0964254b2b4de5acbb850d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:25 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51460
I20260812 02:29:25.280192 15225 tablet_replica.cc:330] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: stopping tablet replica
I20260812 02:29:25.280192 14839 tablet_service.cc:1514] Processing DeleteTablet for tablet c9544c66e0964254b2b4de5acbb850d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:25 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33540
I20260812 02:29:25.280607 15225 raft_consensus.cc:2229] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:25.280720 15226 tablet_replica.cc:330] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: stopping tablet replica
I20260812 02:29:25.280908 15225 raft_consensus.cc:2258] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:25.281154 15226 raft_consensus.cc:2229] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:29:25.281121 15003 tablet_service.cc:1514] Processing DeleteTablet for tablet c9544c66e0964254b2b4de5acbb850d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:25 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:46316
I20260812 02:29:25.281603 15227 tablet_replica.cc:330] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: stopping tablet replica
I20260812 02:29:25.281916 15227 raft_consensus.cc:2229] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:25.282045 15226 raft_consensus.cc:2258] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:25.282178 15227 raft_consensus.cc:2258] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:25.284044 15227 ts_tablet_manager.cc:1905] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:25.286422 15226 ts_tablet_manager.cc:1905] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:25.288682 14503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:25.289608 14503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:25.289613 15225 ts_tablet_manager.cc:1905] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:25.298270 15225 ts_tablet_manager.cc:1918] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:29:25.298557 15225 log.cc:1196] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/c9544c66e0964254b2b4de5acbb850d6
I20260812 02:29:25.299274 15225 ts_tablet_manager.cc:1939] T c9544c66e0964254b2b4de5acbb850d6 P 72504797e7f74fd4b43991a3672fb6e6: Deleting consensus metadata
I20260812 02:29:25.299494 15227 ts_tablet_manager.cc:1918] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:29:25.299913 15227 log.cc:1196] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/c9544c66e0964254b2b4de5acbb850d6
I20260812 02:29:25.300760 15227 ts_tablet_manager.cc:1939] T c9544c66e0964254b2b4de5acbb850d6 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting consensus metadata
I20260812 02:29:25.302080 14671 tablet_service.cc:1467] Processing CreateTablet for tablet f518f43f458a4cd5a95bc1ace7113675 (DEFAULT_TABLE table=test-table [id=041ddf0f53b04d6285aff03d9fd52405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:25.301147 14486 catalog_manager.cc:4841] TS 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887): tablet c9544c66e0964254b2b4de5acbb850d6 (table test-table [id=aabf89ded5bb47ef9683843f8146781e]) successfully deleted
I20260812 02:29:25.302708 14671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f518f43f458a4cd5a95bc1ace7113675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:25.302842 14487 catalog_manager.cc:4841] TS ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797): tablet c9544c66e0964254b2b4de5acbb850d6 (table test-table [id=aabf89ded5bb47ef9683843f8146781e]) successfully deleted
I20260812 02:29:25.302903 15226 ts_tablet_manager.cc:1918] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:29:25.302814 14839 tablet_service.cc:1467] Processing CreateTablet for tablet f518f43f458a4cd5a95bc1ace7113675 (DEFAULT_TABLE table=test-table [id=041ddf0f53b04d6285aff03d9fd52405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:25.303140 15226 log.cc:1196] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/c9544c66e0964254b2b4de5acbb850d6
I20260812 02:29:25.303548 14839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f518f43f458a4cd5a95bc1ace7113675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:25.304033 15226 ts_tablet_manager.cc:1939] T c9544c66e0964254b2b4de5acbb850d6 P 25fd0582c11648059f535cac4b51319a: Deleting consensus metadata
I20260812 02:29:25.305328 14486 catalog_manager.cc:4841] TS 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809): tablet c9544c66e0964254b2b4de5acbb850d6 (table test-table [id=aabf89ded5bb47ef9683843f8146781e]) successfully deleted
I20260812 02:29:25.309836 15003 tablet_service.cc:1467] Processing CreateTablet for tablet f518f43f458a4cd5a95bc1ace7113675 (DEFAULT_TABLE table=test-table [id=041ddf0f53b04d6285aff03d9fd52405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:25.310528 15003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f518f43f458a4cd5a95bc1ace7113675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:25.313592 15240 tablet_bootstrap.cc:492] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: Bootstrap starting.
I20260812 02:29:25.313722 15238 tablet_bootstrap.cc:492] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: Bootstrap starting.
I20260812 02:29:25.316540 15238 tablet_bootstrap.cc:654] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:25.316908 15240 tablet_bootstrap.cc:654] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:25.320559 15236 tablet_bootstrap.cc:492] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: Bootstrap starting.
I20260812 02:29:25.323833 15236 tablet_bootstrap.cc:654] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:25.331663 15236 tablet_bootstrap.cc:492] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: No bootstrap required, opened a new log
I20260812 02:29:25.332008 15240 tablet_bootstrap.cc:492] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: No bootstrap required, opened a new log
I20260812 02:29:25.332018 15236 ts_tablet_manager.cc:1397] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260812 02:29:25.332227 15240 ts_tablet_manager.cc:1397] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260812 02:29:25.333045 15240 raft_consensus.cc:348] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.333346 15240 raft_consensus.cc:374] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:25.333169 15236 raft_consensus.cc:348] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.333433 15240 raft_consensus.cc:729] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72504797e7f74fd4b43991a3672fb6e6, State: Initialized, Role: FOLLOWER
I20260812 02:29:25.333472 15236 raft_consensus.cc:374] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:25.333561 15236 raft_consensus.cc:729] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6aace3e9a743c48b7e766e83bcfc02, State: Initialized, Role: FOLLOWER
I20260812 02:29:25.333706 15240 consensus_queue.cc:260] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.333850 15236 consensus_queue.cc:260] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.334324 15240 ts_tablet_manager.cc:1428] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 02:29:25.337767 15238 tablet_bootstrap.cc:492] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: No bootstrap required, opened a new log
I20260812 02:29:25.338003 15238 ts_tablet_manager.cc:1397] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.015s
I20260812 02:29:25.338961 15238 raft_consensus.cc:348] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.339322 15238 raft_consensus.cc:374] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:25.339517 15238 raft_consensus.cc:729] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fd0582c11648059f535cac4b51319a, State: Initialized, Role: FOLLOWER
I20260812 02:29:25.340890 15238 consensus_queue.cc:260] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.341360 15236 ts_tablet_manager.cc:1428] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260812 02:29:25.343111 15238 ts_tablet_manager.cc:1428] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 02:29:25.358546 15244 raft_consensus.cc:482] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:25.358855 15244 raft_consensus.cc:504] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.359921 15244 leader_election.cc:288] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809), ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
I20260812 02:29:25.361445 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518f43f458a4cd5a95bc1ace7113675" candidate_uuid: "72504797e7f74fd4b43991a3672fb6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fd0582c11648059f535cac4b51319a" is_pre_election: true
I20260812 02:29:25.361773 14862 raft_consensus.cc:2454] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72504797e7f74fd4b43991a3672fb6e6 in term 0.
I20260812 02:29:25.361720 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518f43f458a4cd5a95bc1ace7113675" candidate_uuid: "72504797e7f74fd4b43991a3672fb6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" is_pre_election: true
I20260812 02:29:25.362064 15024 raft_consensus.cc:2454] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72504797e7f74fd4b43991a3672fb6e6 in term 0.
I20260812 02:29:25.362390 14609 leader_election.cc:302] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fd0582c11648059f535cac4b51319a, 72504797e7f74fd4b43991a3672fb6e6; no voters:
I20260812 02:29:25.362975 15244 raft_consensus.cc:2781] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:25.363152 15244 raft_consensus.cc:482] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:25.363269 15244 raft_consensus.cc:3037] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:25.367415 15244 raft_consensus.cc:504] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.368801 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518f43f458a4cd5a95bc1ace7113675" candidate_uuid: "72504797e7f74fd4b43991a3672fb6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fd0582c11648059f535cac4b51319a"
I20260812 02:29:25.369060 14862 raft_consensus.cc:3037] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:25.369433 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518f43f458a4cd5a95bc1ace7113675" candidate_uuid: "72504797e7f74fd4b43991a3672fb6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6aace3e9a743c48b7e766e83bcfc02"
I20260812 02:29:25.369750 15024 raft_consensus.cc:3037] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:25.372397 14862 raft_consensus.cc:2454] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72504797e7f74fd4b43991a3672fb6e6 in term 1.
I20260812 02:29:25.372985 14609 leader_election.cc:302] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fd0582c11648059f535cac4b51319a, 72504797e7f74fd4b43991a3672fb6e6; no voters:
I20260812 02:29:25.374063 15261 raft_consensus.cc:2781] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:25.374163 15024 raft_consensus.cc:2454] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72504797e7f74fd4b43991a3672fb6e6 in term 1.
I20260812 02:29:25.375088 15244 leader_election.cc:288] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [CANDIDATE]: Term 1 election: Requested vote from peers 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809), ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797)
I20260812 02:29:25.375476 15261 raft_consensus.cc:686] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 LEADER]: Becoming Leader. State: Replica: 72504797e7f74fd4b43991a3672fb6e6, State: Running, Role: LEADER
I20260812 02:29:25.376163 15261 consensus_queue.cc:237] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } }
I20260812 02:29:25.380887 14499 catalog_manager.cc:5495] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193). New cstate: current_term: 1 leader_uuid: "72504797e7f74fd4b43991a3672fb6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72504797e7f74fd4b43991a3672fb6e6" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 32887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 } health_report { overall_health: UNKNOWN } } }
I20260812 02:29:25.429699 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:25.422245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.12.8.193:32887)
0812 02:29:25.423587 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:25.423592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:25.423611 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:29:25.423695 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:25.423699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:25.423784 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:25.423870 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:25.424459 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:25.425903 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:25.427355 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:25.428080 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:25.428231 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:25.428656 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:25.428852 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:25.428958 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:25.429070 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 02:29:25.429214 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":1018,"threads_started":1}
I20260812 02:29:25.440840 15024 raft_consensus.cc:1261] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Refusing update from remote peer 72504797e7f74fd4b43991a3672fb6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:25.441352 14862 raft_consensus.cc:1261] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Refusing update from remote peer 72504797e7f74fd4b43991a3672fb6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:25.441630 15244 consensus_queue.cc:1035] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6aace3e9a743c48b7e766e83bcfc02" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:25.442013 15261 consensus_queue.cc:1035] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25fd0582c11648059f535cac4b51319a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:25.447657 15201 mvcc.cc:205] Tried to move back new op lower bound from 7317511231240581120 to 7317511230984757248. Current Snapshot: MvccSnapshot[applied={T|T < 7317511231240581120}]
W20260812 02:29:25.468484 15201 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 02:29:27.104264 14503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:27.104629 14503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:27.110985 14503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f93c3ad5769948cfa73037457ef29fe7: Sending DeleteTablet for 3 replicas of tablet f518f43f458a4cd5a95bc1ace7113675
I20260812 02:29:27.112215 14839 tablet_service.cc:1514] Processing DeleteTablet for tablet f518f43f458a4cd5a95bc1ace7113675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:27 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33540
I20260812 02:29:27.112278 14671 tablet_service.cc:1514] Processing DeleteTablet for tablet f518f43f458a4cd5a95bc1ace7113675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:27 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51460
I20260812 02:29:27.112421 15003 tablet_service.cc:1514] Processing DeleteTablet for tablet f518f43f458a4cd5a95bc1ace7113675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:27 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:46316
I20260812 02:29:27.112980 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14588
I20260812 02:29:27.125641 15301 tablet_replica.cc:330] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: stopping tablet replica
I20260812 02:29:27.126020 15301 raft_consensus.cc:2229] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:29:27.126947 15301 raft_consensus.cc:2258] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:27.128368 15298 tablet_replica.cc:330] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: stopping tablet replica
I20260812 02:29:27.128718 15298 raft_consensus.cc:2229] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:27.129232 15298 raft_consensus.cc:2258] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:27.130118 15302 tablet_replica.cc:330] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: stopping tablet replica
I20260812 02:29:27.130494 15302 raft_consensus.cc:2229] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:27.130719 15298 ts_tablet_manager.cc:1905] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:27.130786 15301 ts_tablet_manager.cc:1905] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:27.131187 15302 raft_consensus.cc:2258] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:27.133268 15302 ts_tablet_manager.cc:1905] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:27.140726 15301 ts_tablet_manager.cc:1918] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:29:27.141026 15301 log.cc:1196] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/f518f43f458a4cd5a95bc1ace7113675
I20260812 02:29:27.141088 15298 ts_tablet_manager.cc:1918] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:29:27.141325 15298 log.cc:1196] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/f518f43f458a4cd5a95bc1ace7113675
I20260812 02:29:27.141892 15301 ts_tablet_manager.cc:1939] T f518f43f458a4cd5a95bc1ace7113675 P 72504797e7f74fd4b43991a3672fb6e6: Deleting consensus metadata
I20260812 02:29:27.142134 15298 ts_tablet_manager.cc:1939] T f518f43f458a4cd5a95bc1ace7113675 P 25fd0582c11648059f535cac4b51319a: Deleting consensus metadata
I20260812 02:29:27.142326 15302 ts_tablet_manager.cc:1918] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:29:27.142587 15302 log.cc:1196] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/f518f43f458a4cd5a95bc1ace7113675
I20260812 02:29:27.143296 15302 ts_tablet_manager.cc:1939] T f518f43f458a4cd5a95bc1ace7113675 P ff6aace3e9a743c48b7e766e83bcfc02: Deleting consensus metadata
I20260812 02:29:27.143693 14486 catalog_manager.cc:4841] TS 25fd0582c11648059f535cac4b51319a (127.12.8.194:42809): tablet f518f43f458a4cd5a95bc1ace7113675 (table test-table [id=041ddf0f53b04d6285aff03d9fd52405]) successfully deleted
I20260812 02:29:27.144187 14486 catalog_manager.cc:4841] TS 72504797e7f74fd4b43991a3672fb6e6 (127.12.8.193:32887): tablet f518f43f458a4cd5a95bc1ace7113675 (table test-table [id=041ddf0f53b04d6285aff03d9fd52405]) successfully deleted
I20260812 02:29:27.144766 14487 catalog_manager.cc:4841] TS ff6aace3e9a743c48b7e766e83bcfc02 (127.12.8.195:41797): tablet f518f43f458a4cd5a95bc1ace7113675 (table test-table [id=041ddf0f53b04d6285aff03d9fd52405]) successfully deleted
Aug 12 02:29:27 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501764, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:27.159629 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:27.114290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.12.8.193:32887)
0812 02:29:27.129523 (+ 15233us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:27.129530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:27.132438 (+ 2908us) server_negotiation.cc:407] Connection header received
0812 02:29:27.132719 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:27.132727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:27.132829 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:27.132942 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:27.136214 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:27.138057 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:27.140622 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:27.142131 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:27.142396 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:27.150313 (+ 7917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:27.150346 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:27.150367 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:27.150427 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:27.155587 (+ 5160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:27.156741 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:27.156750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:27.156753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:27.157005 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:27.157512 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:27.157516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:27.157518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:27.158191 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:27.158543 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:27.158769 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 02:29:27.159012 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14842,"thread_start_us":15062,"threads_started":1}
W20260812 02:29:27.160714 14748 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51488
W20260812 02:29:27.161792 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:27.162138 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 14588
I20260812 02:29:27.198421 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14759
Aug 12 02:29:27 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501764, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:29:27.218061 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:27.199475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.12.8.194:42809)
0812 02:29:27.200421 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:27.200425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:27.200444 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:29:27.200512 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:27.200524 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:27.200613 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:27.200728 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:27.202503 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:27.203387 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:27.204698 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:27.205145 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:27.205288 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:27.210338 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:27.210363 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:27.210384 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:27.210435 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:27.214246 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:27.215297 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:27.215307 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:27.215310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:27.215503 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:27.215949 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:27.215953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:27.215955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:27.216516 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:27.216867 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:27.217315 (+ 448us) server_negotiation.cc:299] Negotiation successful
0812 02:29:27.217616 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":586,"threads_started":1}
W20260812 02:29:27.218919 14914 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33578
W20260812 02:29:27.219718 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14759: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:27.219903 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 14759
I20260812 02:29:27.260195 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14924
Aug 12 02:29:27 dist-test-slave-vq2q krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501764, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:29:27.281440 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:27.261230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.12.8.195:41797)
0812 02:29:27.261970 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:27.261977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:27.261998 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:29:27.262069 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:27.262074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:27.262179 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:27.262302 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:27.264230 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:27.265647 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:27.267051 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:27.267741 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:27.267884 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:27.273562 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:27.273585 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:27.273611 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:27.273658 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:27.278090 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:27.278996 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:27.279007 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:27.279011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:27.279227 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:27.279614 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:27.279619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:27.279623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:27.280161 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:27.280467 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:27.280688 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 02:29:27.280958 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":444,"threads_started":1}
W20260812 02:29:27.282338 15080 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46338
W20260812 02:29:27.283056 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14924: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:27.283257 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 14924
I20260812 02:29:27.316636 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 14464
I20260812 02:29:27.318205 14553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:27.455320 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 14464
2026-08-12T02:29:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8986 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:27 dist-test-slave-vq2q krb5kdc[15311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:27 dist-test-slave-vq2q krb5kdc[15311](info): set up 2 sockets
Aug 12 02:29:27 dist-test-slave-vq2q krb5kdc[15311](info): commencing operation
krb5kdc: starting...
W20260812 02:29:28.625746 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.012s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:28 dist-test-slave-vq2q krb5kdc[15311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:28.860723 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43631
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:43631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:29.211591 15325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:29.211910 15325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:29.212023 15325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:29.219409 15325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:29.219532 15325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:29.219578 15325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:29.219647 15325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:29.219713 15325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:29.230011 15325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:43631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:29.231446 15325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:29.232782 15325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:29.246074 15337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:29.246088 15334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:29.246050 15333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:29.247028 15325 server_base.cc:1034] running on GCE node
I20260812 02:29:29.248132 15325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:29.250262 15325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:29.251554 15325 hybrid_clock.cc:648] HybridClock initialized: now 1786501769251465 us; error 60 us; skew 500 ppm
I20260812 02:29:29.252009 15325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:29 dist-test-slave-vq2q krb5kdc[15311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:29.255851 15325 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:29.257823 15325 webserver.cc:460] Webserver started at http://127.12.8.254:38211/ using document root <none> and password file <none>
I20260812 02:29:29.258734 15325 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:29.258854 15325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:29.259186 15325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:29.263331 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "93da936cdc96423baaea4e9abffe316c"
format_stamp: "Formatted at 2026-08-12 02:29:29 on dist-test-slave-vq2q"
server_key: "5620b3e44da363b41caaa3113ff23b43"
server_key_iv: "5b3d2616fd20b8496422877657c257e3"
server_key_version: "encryptionkey@0"
I20260812 02:29:29.264379 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "93da936cdc96423baaea4e9abffe316c"
format_stamp: "Formatted at 2026-08-12 02:29:29 on dist-test-slave-vq2q"
server_key: "5620b3e44da363b41caaa3113ff23b43"
server_key_iv: "5b3d2616fd20b8496422877657c257e3"
server_key_version: "encryptionkey@0"
I20260812 02:29:29.271634 15325 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 02:29:29.276568 15345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:29.278669 15325 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 02:29:29.278931 15325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "93da936cdc96423baaea4e9abffe316c"
format_stamp: "Formatted at 2026-08-12 02:29:29 on dist-test-slave-vq2q"
server_key: "5620b3e44da363b41caaa3113ff23b43"
server_key_iv: "5b3d2616fd20b8496422877657c257e3"
server_key_version: "encryptionkey@0"
I20260812 02:29:29.279304 15325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:29.300431 15325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:29.305801 15325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:29.306160 15325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:29.328557 15325 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:43631
I20260812 02:29:29.328588 15407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:43631 every 8 connection(s)
I20260812 02:29:29.330425 15325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:29.334863 15408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:29.336735 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 15325
I20260812 02:29:29.336997 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:29.337790 12323 external_mini_cluster.cc:1428] Setting key 7c0a99ce6789499e3680893b15d81169
I20260812 02:29:29.348757 15408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c: Bootstrap starting.
I20260812 02:29:29.352581 15408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:29.353837 15408 log.cc:824] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:29:29 dist-test-slave-vq2q krb5kdc[15311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:29.358119 15408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c: No bootstrap required, opened a new log
I20260812 02:29:29.367611 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:29.340680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.12.8.254:43631)
0812 02:29:29.341853 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:29.341872 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:29.341943 (+ 71us) server_negotiation.cc:407] Connection header received
0812 02:29:29.344376 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:29.344408 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:29.345009 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:29.345542 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:29.347233 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:29.349279 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:29.351040 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:29.352078 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:29.352283 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:29.357682 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:29.357697 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:29.357718 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:29.357782 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:29.362087 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:29.363152 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:29.363161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:29.363169 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:29.363427 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:29.363849 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:29.363855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:29.363858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:29.364696 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:29.365036 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:29.365532 (+ 496us) server_negotiation.cc:299] Negotiation successful
0812 02:29:29.366078 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":545,"threads_started":1}
I20260812 02:29:29.368916 15408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93da936cdc96423baaea4e9abffe316c" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43631 } }
I20260812 02:29:29.369421 15408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:29.369522 15408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93da936cdc96423baaea4e9abffe316c, State: Initialized, Role: FOLLOWER
I20260812 02:29:29.370191 15408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93da936cdc96423baaea4e9abffe316c" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43631 } }
I20260812 02:29:29.370438 15408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:29.370587 15408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:29.370765 15408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:29.373337 15408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93da936cdc96423baaea4e9abffe316c" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43631 } }
I20260812 02:29:29.373867 15408 leader_election.cc:302] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93da936cdc96423baaea4e9abffe316c; no voters:
I20260812 02:29:29.374557 15408 leader_election.cc:288] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:29.374825 15413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:29.375689 15413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [term 1 LEADER]: Becoming Leader. State: Replica: 93da936cdc96423baaea4e9abffe316c, State: Running, Role: LEADER
I20260812 02:29:29.376437 15413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93da936cdc96423baaea4e9abffe316c" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43631 } }
I20260812 02:29:29.376766 15408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:29.386584 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93da936cdc96423baaea4e9abffe316c. Latest consensus state: current_term: 1 leader_uuid: "93da936cdc96423baaea4e9abffe316c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93da936cdc96423baaea4e9abffe316c" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43631 } } }
I20260812 02:29:29.387116 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:29.387528 15414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93da936cdc96423baaea4e9abffe316c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93da936cdc96423baaea4e9abffe316c" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43631 } } }
I20260812 02:29:29.387925 15414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c [sys.catalog]: This master's current role is: LEADER
W20260812 02:29:29.394557 15431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:29:29.394748 15431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:29:29.394922 15433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:29.397452 15433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:29.408818 15433 catalog_manager.cc:1347] Generated new cluster ID: 212e480a9a7b4b7fbea287db730532cf
I20260812 02:29:29.408987 15433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:29.426456 15433 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:29.428151 15433 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:29.438736 15433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93da936cdc96423baaea4e9abffe316c: Generated new TSK 0
I20260812 02:29:29.440166 15433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:29.513651 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43631
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:29.938050 15441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:29.938467 15441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:29.938611 15441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:29.948493 15441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:29.948724 15441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:29.948920 15441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:29.961937 15441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:29.963963 15441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:29.965780 15441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:29.982496 15446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:29.982524 15451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:29.982858 15448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:29.983870 15441 server_base.cc:1034] running on GCE node
I20260812 02:29:29.984717 15441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:29.986163 15441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:29.987435 15441 hybrid_clock.cc:648] HybridClock initialized: now 1786501769987369 us; error 45 us; skew 500 ppm
I20260812 02:29:29.987851 15441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:29 dist-test-slave-vq2q krb5kdc[15311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:29.995584 15441 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:29.997685 15441 webserver.cc:460] Webserver started at http://127.12.8.193:32963/ using document root <none> and password file <none>
I20260812 02:29:29.998622 15441 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:29.998757 15441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:29.999130 15441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:30.002873 15441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "67b142b3458c48b3a7ce2d0cf040e35a"
format_stamp: "Formatted at 2026-08-12 02:29:29 on dist-test-slave-vq2q"
server_key: "6a3dd0a12ef1eb861cd1e9a6ff8974f9"
server_key_iv: "18e8c79e6d7dd8f53645fab94807719c"
server_key_version: "encryptionkey@0"
I20260812 02:29:30.003871 15441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "67b142b3458c48b3a7ce2d0cf040e35a"
format_stamp: "Formatted at 2026-08-12 02:29:29 on dist-test-slave-vq2q"
server_key: "6a3dd0a12ef1eb861cd1e9a6ff8974f9"
server_key_iv: "18e8c79e6d7dd8f53645fab94807719c"
server_key_version: "encryptionkey@0"
I20260812 02:29:30.011053 15441 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260812 02:29:30.016356 15461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:30.018565 15441 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 02:29:30.018847 15441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "67b142b3458c48b3a7ce2d0cf040e35a"
format_stamp: "Formatted at 2026-08-12 02:29:29 on dist-test-slave-vq2q"
server_key: "6a3dd0a12ef1eb861cd1e9a6ff8974f9"
server_key_iv: "18e8c79e6d7dd8f53645fab94807719c"
server_key_version: "encryptionkey@0"
I20260812 02:29:30.019119 15441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:30.037117 15441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:30.041779 15441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:30.042100 15441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:30.043277 15441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:30.044905 15441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:30.044988 15441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:30.045121 15441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:30.045209 15441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:30.082360 15441 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:32985
I20260812 02:29:30.082520 15596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:32985 every 8 connection(s)
I20260812 02:29:30.083948 15441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:30.094410 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 15441
I20260812 02:29:30.094688 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:30.095558 12323 external_mini_cluster.cc:1428] Setting key 4017fa8b04dbc1ac36fbc38cd5a35ed3
Aug 12 02:29:30 dist-test-slave-vq2q krb5kdc[15311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501769, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:30.111125 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:30.087479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:43239 (local address 127.12.8.254:43631)
0812 02:29:30.088094 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:30.088100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:30.089153 (+ 1053us) server_negotiation.cc:407] Connection header received
0812 02:29:30.091595 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:30.091601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:30.091729 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:30.091855 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:30.094227 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:30.095165 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:30.097666 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:30.098233 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:30.098377 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:30.104145 (+ 5768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:30.104150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:30.104154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:30.104193 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:30.106865 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:30.108097 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:30.108101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:30.108103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:30.108246 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:30.108851 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:30.108854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:30.108856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:30.109152 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:30.109322 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:30.110442 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0812 02:29:30.110698 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":283,"threads_started":1}
I20260812 02:29:30.112313 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:30.087892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43631 (local address 127.12.8.193:43239)
0812 02:29:30.088886 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:30.088946 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:30.091221 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:30.092191 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:30.092201 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:30.092933 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:30.093869 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:30.093906 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:30.095438 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:30.095446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:30.097458 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:30.097488 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:30.098602 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:30.098614 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:30.099023 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:30.100003 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:30.100046 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:30.103769 (+ 3723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:30.107082 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:30.107095 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:30.107122 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:30.107917 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:30.108365 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:30.108370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:30.108373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:30.108697 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:30.109322 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:30.109332 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:30.109947 (+ 615us) client_negotiation.cc:769] Sending connection context
0812 02:29:30.110286 (+ 339us) client_negotiation.cc:240] Negotiation successful
0812 02:29:30.110843 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":374,"threads_started":1}
I20260812 02:29:30.114480 15597 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43631
I20260812 02:29:30.114876 15597 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:30.115834 15597 heartbeater.cc:511] Master 127.12.8.254:43631 requested a full tablet report, sending...
I20260812 02:29:30.118839 15366 ts_manager.cc:194] Registered new tserver with Master: 67b142b3458c48b3a7ce2d0cf040e35a (127.12.8.193:32985)
I20260812 02:29:30.121119 15366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:43239
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:30.157274 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43631
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:30.522940 15604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:30.523228 15604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:30.523350 15604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:30.530911 15604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:30.531037 15604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:30.531184 15604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:30.541162 15604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:30.542589 15604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:30.543926 15604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:30.558017 15612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:30.558141 15609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:30.558116 15610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:30.559147 15604 server_base.cc:1034] running on GCE node
I20260812 02:29:30.559832 15604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:30.560838 15604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:30.562088 15604 hybrid_clock.cc:648] HybridClock initialized: now 1786501770562026 us; error 58 us; skew 500 ppm
I20260812 02:29:30.562444 15604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:30 dist-test-slave-vq2q krb5kdc[15311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501770, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:30.565948 15604 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:30.567890 15604 webserver.cc:460] Webserver started at http://127.12.8.194:33979/ using document root <none> and password file <none>
I20260812 02:29:30.568819 15604 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:30.568967 15604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:30.569394 15604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:30.573025 15604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "4d756783204f4143a8db9c62c4dee403"
format_stamp: "Formatted at 2026-08-12 02:29:30 on dist-test-slave-vq2q"
server_key: "f54bb3b928cf40c04ea30d4188246598"
server_key_iv: "628cd4c83375112c2dffe3fe6110606d"
server_key_version: "encryptionkey@0"
I20260812 02:29:30.574049 15604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "4d756783204f4143a8db9c62c4dee403"
format_stamp: "Formatted at 2026-08-12 02:29:30 on dist-test-slave-vq2q"
server_key: "f54bb3b928cf40c04ea30d4188246598"
server_key_iv: "628cd4c83375112c2dffe3fe6110606d"
server_key_version: "encryptionkey@0"
I20260812 02:29:30.581009 15604 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 02:29:30.585660 15623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:30.587466 15604 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:30.587703 15604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "4d756783204f4143a8db9c62c4dee403"
format_stamp: "Formatted at 2026-08-12 02:29:30 on dist-test-slave-vq2q"
server_key: "f54bb3b928cf40c04ea30d4188246598"
server_key_iv: "628cd4c83375112c2dffe3fe6110606d"
server_key_version: "encryptionkey@0"
I20260812 02:29:30.587946 15604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:30.609022 15604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:30.614251 15604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:30.614598 15604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:30.615820 15604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:30.617655 15604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:30.617743 15604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:30.617861 15604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:30.617909 15604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:30.655911 15604 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:43145
I20260812 02:29:30.656036 15759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:43145 every 8 connection(s)
I20260812 02:29:30.657658 15604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:30.667711 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 15604
I20260812 02:29:30.667979 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:30.668628 12323 external_mini_cluster.cc:1428] Setting key df61999302e56aea6489276ba20e4fb2
Aug 12 02:29:30 dist-test-slave-vq2q krb5kdc[15311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501770, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:30.684266 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:30.660904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:57015 (local address 127.12.8.254:43631)
0812 02:29:30.661506 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:30.661514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:30.662827 (+ 1313us) server_negotiation.cc:407] Connection header received
0812 02:29:30.664906 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:30.664912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:30.665026 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:30.665251 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:30.667219 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:30.668385 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:30.670339 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:30.670892 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:30.671042 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:30.676735 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:30.676742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:30.676747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:30.676796 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:30.679553 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:30.680833 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:30.680838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:30.680841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:30.680989 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:30.681710 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:30.681717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:30.681720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:30.682223 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:30.682393 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:30.683424 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0812 02:29:30.683816 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":291,"threads_started":1}
I20260812 02:29:30.685129 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:30.661643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43631 (local address 127.12.8.194:57015)
0812 02:29:30.662621 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:30.662676 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:30.664565 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:30.665491 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:30.665500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:30.666133 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:30.666806 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:30.666825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:30.668607 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:30.668612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:30.670146 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:30.670160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:30.671271 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:30.671283 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:30.671712 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:30.672666 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:30.672709 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:30.676290 (+ 3581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:30.679778 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:30.679792 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:30.679818 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:30.680612 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:30.681121 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:30.681125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:30.681128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:30.681507 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:30.682459 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:30.682469 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:30.682924 (+ 455us) client_negotiation.cc:769] Sending connection context
0812 02:29:30.683206 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 02:29:30.683776 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":373,"threads_started":1}
I20260812 02:29:30.687107 15760 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43631
I20260812 02:29:30.687546 15760 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:30.688490 15760 heartbeater.cc:511] Master 127.12.8.254:43631 requested a full tablet report, sending...
I20260812 02:29:30.690524 15366 ts_manager.cc:194] Registered new tserver with Master: 4d756783204f4143a8db9c62c4dee403 (127.12.8.194:43145)
I20260812 02:29:30.691570 15366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:57015
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:30.733309 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43631
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:31.084424 15767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:31.084853 15767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:31.084997 15767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:31.092918 15767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:31.093057 15767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:31.093247 15767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:31.104005 15767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:31.105504 15767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:31.106938 15767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:31.121307 15774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:31.121411 15772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:31.123208 15776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:31.124081 15767 server_base.cc:1034] running on GCE node
I20260812 02:29:31.124791 15767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:31.125538 15597 heartbeater.cc:503] Master 127.12.8.254:43631 was elected leader, sending a full tablet report...
I20260812 02:29:31.126060 15767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:31.127346 15767 hybrid_clock.cc:648] HybridClock initialized: now 1786501771127281 us; error 56 us; skew 500 ppm
I20260812 02:29:31.127774 15767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:31 dist-test-slave-vq2q krb5kdc[15311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501771, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:31.131593 15767 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:29:31.133671 15767 webserver.cc:460] Webserver started at http://127.12.8.195:46209/ using document root <none> and password file <none>
I20260812 02:29:31.134616 15767 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:31.134755 15767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:31.135133 15767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:31.138906 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "91d8643da7fa491e8534a9f7d3f44a8c"
format_stamp: "Formatted at 2026-08-12 02:29:31 on dist-test-slave-vq2q"
server_key: "2a23727fe71d8781d65b61d27136704e"
server_key_iv: "2482055717b54b9788a9c8a0d07b7c9a"
server_key_version: "encryptionkey@0"
I20260812 02:29:31.139868 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "91d8643da7fa491e8534a9f7d3f44a8c"
format_stamp: "Formatted at 2026-08-12 02:29:31 on dist-test-slave-vq2q"
server_key: "2a23727fe71d8781d65b61d27136704e"
server_key_iv: "2482055717b54b9788a9c8a0d07b7c9a"
server_key_version: "encryptionkey@0"
I20260812 02:29:31.146715 15767 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 02:29:31.151566 15783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:31.153680 15767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:29:31.153913 15767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "91d8643da7fa491e8534a9f7d3f44a8c"
format_stamp: "Formatted at 2026-08-12 02:29:31 on dist-test-slave-vq2q"
server_key: "2a23727fe71d8781d65b61d27136704e"
server_key_iv: "2482055717b54b9788a9c8a0d07b7c9a"
server_key_version: "encryptionkey@0"
I20260812 02:29:31.154153 15767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:31.178862 15767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:31.184252 15767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:31.184605 15767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:31.186659 15767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:31.188349 15767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:31.188442 15767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:31.188570 15767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:31.188629 15767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:31.225870 15767 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:42203
I20260812 02:29:31.225957 15914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:42203 every 8 connection(s)
I20260812 02:29:31.227267 15767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:31.233424 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 15767
I20260812 02:29:31.233649 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:31.234151 12323 external_mini_cluster.cc:1428] Setting key 00095855cd37adabfc714bf85b1c5a64
Aug 12 02:29:31 dist-test-slave-vq2q krb5kdc[15311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501771, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:31.251125 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:31.230580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:58635 (local address 127.12.8.254:43631)
0812 02:29:31.231228 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:31.231236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:31.232472 (+ 1236us) server_negotiation.cc:407] Connection header received
0812 02:29:31.234601 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:31.234608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:31.234763 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:31.234905 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:31.236857 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:31.237926 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:31.239554 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:31.239982 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:31.240096 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:31.243705 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:31.243711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:31.243715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:31.243753 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:31.246868 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:31.247751 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:31.247755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:31.247758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:31.247962 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:31.248614 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:31.248618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:31.248620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:31.249044 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:31.249302 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:31.250384 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0812 02:29:31.250714 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":277,"threads_started":1}
I20260812 02:29:31.252004 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:31.231289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43631 (local address 127.12.8.195:58635)
0812 02:29:31.232239 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:31.232295 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:31.234245 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:31.235209 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:31.235219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:31.235900 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:31.236634 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:31.236655 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:31.238077 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:31.238081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:31.239401 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:31.239415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:31.240227 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:31.240234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:31.240471 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:31.241172 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:31.241233 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:31.243463 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:31.247059 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:31.247066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:31.247081 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:31.247605 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:31.248139 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:31.248144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:31.248147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:31.248474 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:31.249284 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:31.249294 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:31.249829 (+ 535us) client_negotiation.cc:769] Sending connection context
0812 02:29:31.250223 (+ 394us) client_negotiation.cc:240] Negotiation successful
0812 02:29:31.250741 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":373,"threads_started":1}
I20260812 02:29:31.253837 15916 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43631
I20260812 02:29:31.254241 15916 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:31.255259 15916 heartbeater.cc:511] Master 127.12.8.254:43631 requested a full tablet report, sending...
I20260812 02:29:31.257258 15366 ts_manager.cc:194] Registered new tserver with Master: 91d8643da7fa491e8534a9f7d3f44a8c (127.12.8.195:42203)
I20260812 02:29:31.258229 15366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:58635
I20260812 02:29:31.263739 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:29:31.311537 15918 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:29:31.308303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.12.8.254:43631)
0812 02:29:31.308636 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:31.308642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:31.308965 (+ 323us) server_negotiation.cc:407] Connection header received
0812 02:29:31.309161 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:31.309165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:31.309300 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:31.309424 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:31.311200 (+ 1776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56742: BlockingRecv error: recv got EOF from 127.0.0.1:56742 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
I20260812 02:29:31.316061 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 15441
W20260812 02:29:31.319392 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 15441: Not authorized: Client connection negotiation failed: client connection to 127.12.8.193:32985: server requires authentication, but client does not have Kerberos credentials available
I20260812 02:29:31.319593 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 15441
I20260812 02:29:31.351521 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 15604
W20260812 02:29:31.354960 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 15604: Not authorized: Client connection negotiation failed: client connection to 127.12.8.194:43145: server requires authentication, but client does not have Kerberos credentials available
I20260812 02:29:31.355196 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 15604
I20260812 02:29:31.381696 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 15767
W20260812 02:29:31.385640 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 15767: Not authorized: Client connection negotiation failed: client connection to 127.12.8.195:42203: server requires authentication, but client does not have Kerberos credentials available
I20260812 02:29:31.385897 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 15767
I20260812 02:29:31.409355 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 15325
I20260812 02:29:31.411463 15403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:31.529834 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 15325
2026-08-12T02:29:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4066 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:31 dist-test-slave-vq2q krb5kdc[15939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:31 dist-test-slave-vq2q krb5kdc[15939](info): set up 2 sockets
Aug 12 02:29:31 dist-test-slave-vq2q krb5kdc[15939](info): commencing operation
krb5kdc: starting...
W20260812 02:29:32.697533 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:33 dist-test-slave-vq2q krb5kdc[15939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260812 02:29:33.054644 12323 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.111s user 0.000s sys 0.050s
2026-08-12T02:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:33Z Disabled control of system clock
W20260812 02:29:33.165329 12323 mini_chronyd.cc:189] Time spent starting chronyd: real 0.108s user 0.000s sys 0.061s
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
W20260812 02:29:33.293856 12323 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.8.254: real 0.126s user 0.002s sys 0.047s
I20260812 02:29:33.296451 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:36463
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:36463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:34.003589 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:34.003918 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:34.004055 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:34.011577 15956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:34.011704 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:34.011785 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:34.011865 15956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:34.011933 15956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:34.022271 15956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:36463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:36463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:34.023682 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:34.025046 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:34.038013 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:34.038074 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:34.038074 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:34.039506 15956 server_base.cc:1034] running on GCE node
I20260812 02:29:34.040406 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:34.042053 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:34.043280 15956 hybrid_clock.cc:648] HybridClock initialized: now 1786501774043214 us; error 72 us; skew 500 ppm
I20260812 02:29:34.043675 15956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:34 dist-test-slave-vq2q krb5kdc[15939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501774, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:34.047235 15956 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:34.049024 15956 webserver.cc:460] Webserver started at http://127.12.8.254:39131/ using document root <none> and password file <none>
I20260812 02:29:34.049827 15956 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:34.049954 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:34.050268 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:34.053506 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14"
format_stamp: "Formatted at 2026-08-12 02:29:34 on dist-test-slave-vq2q"
server_key: "51943ab27ab2a94a7ac4ac57c9e8ccd4"
server_key_iv: "64273db2e9e37849feac0fed5dca7107"
server_key_version: "encryptionkey@0"
I20260812 02:29:34.054409 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14"
format_stamp: "Formatted at 2026-08-12 02:29:34 on dist-test-slave-vq2q"
server_key: "51943ab27ab2a94a7ac4ac57c9e8ccd4"
server_key_iv: "64273db2e9e37849feac0fed5dca7107"
server_key_version: "encryptionkey@0"
I20260812 02:29:34.060500 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 02:29:34.064932 15971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:34.066717 15956 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 02:29:34.066956 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14"
format_stamp: "Formatted at 2026-08-12 02:29:34 on dist-test-slave-vq2q"
server_key: "51943ab27ab2a94a7ac4ac57c9e8ccd4"
server_key_iv: "64273db2e9e37849feac0fed5dca7107"
server_key_version: "encryptionkey@0"
I20260812 02:29:34.067154 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:34.080720 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:34.085408 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:34.085697 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:34.106930 15956 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:36463
I20260812 02:29:34.107059 16037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:36463 every 8 connection(s)
I20260812 02:29:34.108395 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:34.113686 16038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:34.113644 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 15956
I20260812 02:29:34.113935 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:34.114885 12323 external_mini_cluster.cc:1428] Setting key 7bbe10985098836050ee867de3c2e6fe
I20260812 02:29:34.131440 16038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14: Bootstrap starting.
Aug 12 02:29:34 dist-test-slave-vq2q krb5kdc[15939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:34.135792 16038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:34.137161 16038 log.cc:824] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:34.141469 16038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14: No bootstrap required, opened a new log
I20260812 02:29:34.142530 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:34.117340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.12.8.254:36463)
0812 02:29:34.118446 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:34.118459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:34.118516 (+ 57us) server_negotiation.cc:407] Connection header received
0812 02:29:34.121517 (+ 3001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:34.121546 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:34.122053 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:34.122476 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:34.124739 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:34.126158 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:34.128152 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:34.128905 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:34.129049 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:34.134523 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:34.134532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:34.134549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:34.134592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:34.137731 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:34.138743 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:34.138750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:34.138757 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:34.138951 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:34.139435 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:34.139438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:34.139440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:34.140031 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:34.140265 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:34.140895 (+ 630us) server_negotiation.cc:299] Negotiation successful
0812 02:29:34.141334 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":565,"threads_started":1}
I20260812 02:29:34.147553 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:29:34.151517 16038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36463 } }
I20260812 02:29:34.152019 16038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:34.152122 16038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c39a4e6539c41e6b2aeaad5fdba3b14, State: Initialized, Role: FOLLOWER
I20260812 02:29:34.152830 16038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36463 } }
I20260812 02:29:34.153090 16038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:34.153288 16038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:34.153473 16038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:34.156055 16038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36463 } }
I20260812 02:29:34.156752 16038 leader_election.cc:302] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c39a4e6539c41e6b2aeaad5fdba3b14; no voters:
I20260812 02:29:34.157531 16038 leader_election.cc:288] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:34.157660 16044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:34.158986 16044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [term 1 LEADER]: Becoming Leader. State: Replica: 3c39a4e6539c41e6b2aeaad5fdba3b14, State: Running, Role: LEADER
I20260812 02:29:34.159719 16044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36463 } }
I20260812 02:29:34.160446 16038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:34.163236 16047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36463 } } }
I20260812 02:29:34.163954 16047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:34.165752 16048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c39a4e6539c41e6b2aeaad5fdba3b14. Latest consensus state: current_term: 1 leader_uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c39a4e6539c41e6b2aeaad5fdba3b14" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36463 } } }
I20260812 02:29:34.166282 16048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:34.170336 16051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:34.181417 16051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:34.203289 16051 catalog_manager.cc:1347] Generated new cluster ID: d19afc254f1b4d7e95e752f1e1cbed01
I20260812 02:29:34.203452 16051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:34.220346 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:34.209151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.12.8.254:36463)
0812 02:29:34.209373 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:34.209379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:34.210131 (+ 752us) server_negotiation.cc:407] Connection header received
0812 02:29:34.210382 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:34.210389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:34.210505 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:34.210645 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:34.212079 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:34.213349 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:34.215344 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:34.216146 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:34.216321 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:34.217762 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:34.217771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:34.217777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:34.219235 (+ 1458us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:34.219343 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:34.219393 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:34.219526 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:34.219697 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 02:29:34.219845 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 02:29:34.229615 16051 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:34.231755 16051 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:34.246533 16051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c39a4e6539c41e6b2aeaad5fdba3b14: Generated new TSK 0
I20260812 02:29:34.247758 16051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:29:34.259807 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 15956
I20260812 02:29:34.261420 16031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:34.356910 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 15956
2026-08-12T02:29:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2824 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:34 dist-test-slave-vq2q krb5kdc[16079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:34 dist-test-slave-vq2q krb5kdc[16079](info): set up 2 sockets
Aug 12 02:29:34 dist-test-slave-vq2q krb5kdc[16079](info): commencing operation
krb5kdc: starting...
W20260812 02:29:35.512899 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:35 dist-test-slave-vq2q krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:35.794014 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40941
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:36.211921 16093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:36.212257 16093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:36.212387 16093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:36.219656 16093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:36.219791 16093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:36.219862 16093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:36.219947 16093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:36.220011 16093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:36.230158 16093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:36.231652 16093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:36.233105 16093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:36.246471 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:36.246829 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:36.246781 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:36.248106 16093 server_base.cc:1034] running on GCE node
I20260812 02:29:36.249051 16093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:36.250777 16093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:36.252013 16093 hybrid_clock.cc:648] HybridClock initialized: now 1786501776251938 us; error 72 us; skew 500 ppm
I20260812 02:29:36.252395 16093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:36 dist-test-slave-vq2q krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501776, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:36.255895 16093 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:36.257789 16093 webserver.cc:460] Webserver started at http://127.12.8.254:44449/ using document root <none> and password file <none>
I20260812 02:29:36.258560 16093 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:36.258677 16093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:36.259029 16093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:36.262378 16093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "a5ae3c462bfa4db7b30cac90ea173e31"
format_stamp: "Formatted at 2026-08-12 02:29:36 on dist-test-slave-vq2q"
server_key: "77f5870244b70e93352df3d70b76288e"
server_key_iv: "36251e763ea95ce0e645a341c3ed23f8"
server_key_version: "encryptionkey@0"
I20260812 02:29:36.263157 16093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "a5ae3c462bfa4db7b30cac90ea173e31"
format_stamp: "Formatted at 2026-08-12 02:29:36 on dist-test-slave-vq2q"
server_key: "77f5870244b70e93352df3d70b76288e"
server_key_iv: "36251e763ea95ce0e645a341c3ed23f8"
server_key_version: "encryptionkey@0"
I20260812 02:29:36.269261 16093 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 02:29:36.273648 16116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:36.275529 16093 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:36.275736 16093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "a5ae3c462bfa4db7b30cac90ea173e31"
format_stamp: "Formatted at 2026-08-12 02:29:36 on dist-test-slave-vq2q"
server_key: "77f5870244b70e93352df3d70b76288e"
server_key_iv: "36251e763ea95ce0e645a341c3ed23f8"
server_key_version: "encryptionkey@0"
I20260812 02:29:36.275946 16093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:36.304399 16093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:36.308837 16093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:36.309115 16093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:36.329123 16093 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:40941
I20260812 02:29:36.329246 16181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:40941 every 8 connection(s)
I20260812 02:29:36.330706 16093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:36.335635 16182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:36.335738 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 16093
I20260812 02:29:36.336043 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:36.336898 12323 external_mini_cluster.cc:1428] Setting key 5ddfad286e9d24b91f07d9fd215c02a4
I20260812 02:29:36.353199 16182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31: Bootstrap starting.
Aug 12 02:29:36 dist-test-slave-vq2q krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:36.357761 16182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:36.359122 16182 log.cc:824] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:36.362879 16182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31: No bootstrap required, opened a new log
I20260812 02:29:36.364332 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:36.339513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.12.8.254:40941)
0812 02:29:36.340459 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:36.340474 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:36.340525 (+ 51us) server_negotiation.cc:407] Connection header received
0812 02:29:36.343235 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:36.343264 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:36.343746 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:36.344173 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:36.346322 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:36.347725 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:36.349698 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:36.350465 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:36.350621 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:36.356397 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:36.356408 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:36.356426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:36.356481 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:36.359801 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:36.360602 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:36.360609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:36.360616 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:36.360818 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:36.361220 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:36.361233 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:36.361236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:36.361848 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:36.362172 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:36.362646 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 02:29:36.363136 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":426,"threads_started":1}
I20260812 02:29:36.372330 16182 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ae3c462bfa4db7b30cac90ea173e31" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40941 } }
I20260812 02:29:36.372799 16182 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:36.372900 16182 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ae3c462bfa4db7b30cac90ea173e31, State: Initialized, Role: FOLLOWER
I20260812 02:29:36.373624 16182 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ae3c462bfa4db7b30cac90ea173e31" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40941 } }
I20260812 02:29:36.373901 16182 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:36.374042 16182 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:36.374217 16182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:36.376935 16182 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ae3c462bfa4db7b30cac90ea173e31" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40941 } }
I20260812 02:29:36.377645 16182 leader_election.cc:302] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5ae3c462bfa4db7b30cac90ea173e31; no voters:
I20260812 02:29:36.378419 16182 leader_election.cc:288] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:36.378614 16188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:36.379534 16188 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [term 1 LEADER]: Becoming Leader. State: Replica: a5ae3c462bfa4db7b30cac90ea173e31, State: Running, Role: LEADER
I20260812 02:29:36.380219 16188 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ae3c462bfa4db7b30cac90ea173e31" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40941 } }
I20260812 02:29:36.381235 16182 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:36.383641 16189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5ae3c462bfa4db7b30cac90ea173e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ae3c462bfa4db7b30cac90ea173e31" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40941 } } }
I20260812 02:29:36.384174 16189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:36.384704 16191 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5ae3c462bfa4db7b30cac90ea173e31. Latest consensus state: current_term: 1 leader_uuid: "a5ae3c462bfa4db7b30cac90ea173e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ae3c462bfa4db7b30cac90ea173e31" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40941 } } }
I20260812 02:29:36.385053 16191 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:36.388944 16195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:36.394797 16195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:36.406040 16195 catalog_manager.cc:1347] Generated new cluster ID: d23bcbaf9e584b07b60a842bb0e6e376
I20260812 02:29:36.406306 16195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:36.422003 16195 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:36.423766 16195 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:36.436908 16195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5ae3c462bfa4db7b30cac90ea173e31: Generated new TSK 0
I20260812 02:29:36.438112 16195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:36.518231 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40941
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:36.948601 16218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:36.948987 16218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:36.949131 16218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:36.957396 16218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:36.957561 16218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:36.957726 16218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:36.969043 16218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:36.970950 16218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:36.972648 16218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:36.989321 16224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:36.989153 16225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:36.989351 16229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:36.990723 16218 server_base.cc:1034] running on GCE node
I20260812 02:29:36.991608 16218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:36.992894 16218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:36.994211 16218 hybrid_clock.cc:648] HybridClock initialized: now 1786501776994113 us; error 74 us; skew 500 ppm
I20260812 02:29:36.994820 16218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:36 dist-test-slave-vq2q krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501776, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:36.999189 16218 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:37.001101 16218 webserver.cc:460] Webserver started at http://127.12.8.193:36519/ using document root <none> and password file <none>
I20260812 02:29:37.001947 16218 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:37.002087 16218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:37.002446 16218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:37.005882 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "e81a832b7e394b6487e0bc148150ae1b"
format_stamp: "Formatted at 2026-08-12 02:29:36 on dist-test-slave-vq2q"
server_key: "fbfb77c2132775df1ffb37dd1a7a8ed3"
server_key_iv: "b75f179715fd8407eabdce31ba728eff"
server_key_version: "encryptionkey@0"
I20260812 02:29:37.006731 16218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "e81a832b7e394b6487e0bc148150ae1b"
format_stamp: "Formatted at 2026-08-12 02:29:36 on dist-test-slave-vq2q"
server_key: "fbfb77c2132775df1ffb37dd1a7a8ed3"
server_key_iv: "b75f179715fd8407eabdce31ba728eff"
server_key_version: "encryptionkey@0"
I20260812 02:29:37.013516 16218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 02:29:37.018705 16239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:37.020753 16218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:29:37.021004 16218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "e81a832b7e394b6487e0bc148150ae1b"
format_stamp: "Formatted at 2026-08-12 02:29:36 on dist-test-slave-vq2q"
server_key: "fbfb77c2132775df1ffb37dd1a7a8ed3"
server_key_iv: "b75f179715fd8407eabdce31ba728eff"
server_key_version: "encryptionkey@0"
I20260812 02:29:37.021268 16218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:37.050638 16218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:37.055434 16218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:37.055778 16218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:37.056915 16218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:37.058543 16218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:37.058627 16218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:37.058784 16218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:37.058852 16218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:37.096704 16218 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:39701
I20260812 02:29:37.096810 16377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:39701 every 8 connection(s)
I20260812 02:29:37.098132 16218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:37.101940 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 16218
I20260812 02:29:37.102250 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:37.103192 12323 external_mini_cluster.cc:1428] Setting key d1d15de8390d5ff535d11df73050a4f9
Aug 12 02:29:37 dist-test-slave-vq2q krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501776, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:37.119429 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:37.101453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:34729 (local address 127.12.8.254:40941)
0812 02:29:37.101935 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:37.101940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:37.103542 (+ 1602us) server_negotiation.cc:407] Connection header received
0812 02:29:37.105867 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:37.105873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:37.105957 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:37.106054 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:37.107683 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:37.108411 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:37.109996 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:37.110400 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:37.110519 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:37.113957 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:37.113962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:37.113965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:37.113995 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:37.116384 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:37.117078 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:37.117082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:37.117084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:37.117245 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:37.117673 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:37.117678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:37.117681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:37.117950 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:37.118096 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:37.118810 (+ 714us) server_negotiation.cc:299] Negotiation successful
0812 02:29:37.119042 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":271,"threads_started":1}
I20260812 02:29:37.120457 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:37.102308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40941 (local address 127.12.8.193:34729)
0812 02:29:37.103364 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:37.103414 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:37.105497 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:37.106341 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:37.106350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:37.106893 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:37.107506 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:37.107524 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:37.108562 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:37.108566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:37.109836 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:37.109855 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:37.110633 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:37.110640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:37.110866 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:37.111502 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:37.111525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:37.113747 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:37.116518 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:37.116527 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:37.116543 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:37.116953 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:37.117349 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:37.117354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:37.117359 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:37.117554 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:37.118079 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:37.118086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:37.118420 (+ 334us) client_negotiation.cc:769] Sending connection context
0812 02:29:37.118676 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 02:29:37.119212 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":433,"threads_started":1}
I20260812 02:29:37.122478 16379 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40941
I20260812 02:29:37.122876 16379 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:37.123984 16379 heartbeater.cc:511] Master 127.12.8.254:40941 requested a full tablet report, sending...
I20260812 02:29:37.126605 16140 ts_manager.cc:194] Registered new tserver with Master: e81a832b7e394b6487e0bc148150ae1b (127.12.8.193:39701)
I20260812 02:29:37.128149 16140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:34729
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:37.169780 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40941
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:37.592995 16388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:37.593412 16388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:37.593576 16388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:37.601974 16388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:37.602139 16388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:37.602315 16388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:37.613914 16388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:37.615736 16388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:37.617449 16388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:37.634804 16397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:37.635684 16393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:37.635840 16394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:37.636327 16388 server_base.cc:1034] running on GCE node
I20260812 02:29:37.637341 16388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:37.638690 16388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:37.639959 16388 hybrid_clock.cc:648] HybridClock initialized: now 1786501777639885 us; error 58 us; skew 500 ppm
I20260812 02:29:37.640381 16388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:37 dist-test-slave-vq2q krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501777, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:37.644403 16388 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:37.646553 16388 webserver.cc:460] Webserver started at http://127.12.8.194:42655/ using document root <none> and password file <none>
I20260812 02:29:37.647471 16388 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:37.647619 16388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:37.647994 16388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:37.652025 16388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "6aef9b40832a48e98a6a195584eed96a"
format_stamp: "Formatted at 2026-08-12 02:29:37 on dist-test-slave-vq2q"
server_key: "5b9f0faf9c1575c489665aa568b4bece"
server_key_iv: "4ca3466895e03cae60d76a966da34d52"
server_key_version: "encryptionkey@0"
I20260812 02:29:37.653056 16388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "6aef9b40832a48e98a6a195584eed96a"
format_stamp: "Formatted at 2026-08-12 02:29:37 on dist-test-slave-vq2q"
server_key: "5b9f0faf9c1575c489665aa568b4bece"
server_key_iv: "4ca3466895e03cae60d76a966da34d52"
server_key_version: "encryptionkey@0"
I20260812 02:29:37.660171 16388 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 02:29:37.665099 16405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:37.667029 16388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 02:29:37.667287 16388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "6aef9b40832a48e98a6a195584eed96a"
format_stamp: "Formatted at 2026-08-12 02:29:37 on dist-test-slave-vq2q"
server_key: "5b9f0faf9c1575c489665aa568b4bece"
server_key_iv: "4ca3466895e03cae60d76a966da34d52"
server_key_version: "encryptionkey@0"
I20260812 02:29:37.667578 16388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:37.706415 16388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:37.711984 16388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:37.712354 16388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:37.713716 16388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:37.715508 16388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:37.715600 16388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:37.715735 16388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:37.715794 16388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:37.757081 16388 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:37539
I20260812 02:29:37.757148 16542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:37539 every 8 connection(s)
I20260812 02:29:37.758908 16388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:37.760905 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 16388
I20260812 02:29:37.761150 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:37.761720 12323 external_mini_cluster.cc:1428] Setting key 71b52585b63f5feea34c708f429e94e4
Aug 12 02:29:37 dist-test-slave-vq2q krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501777, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:37.787031 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:37.762463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:56933 (local address 127.12.8.254:40941)
0812 02:29:37.763085 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:37.763090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:37.764479 (+ 1389us) server_negotiation.cc:407] Connection header received
0812 02:29:37.767307 (+ 2828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:37.767313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:37.767399 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:37.767542 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:37.769802 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:37.770660 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:37.772891 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:37.773518 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:37.773634 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:37.778432 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:37.778441 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:37.778447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:37.778506 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:37.782439 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:37.783815 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:37.783822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:37.783825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:37.784002 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:37.784710 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:37.784715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:37.784718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:37.785221 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:37.785389 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:37.786261 (+ 872us) server_negotiation.cc:299] Negotiation successful
0812 02:29:37.786568 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":372,"threads_started":1}
I20260812 02:29:37.787920 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:37.763228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40941 (local address 127.12.8.194:56933)
0812 02:29:37.764236 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:37.764333 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:37.766930 (+ 2597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:37.767858 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:37.767871 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:37.768642 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:37.769574 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:37.769600 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:37.770850 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:37.770855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:37.772727 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:37.772743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:37.773773 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:37.773781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:37.774076 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:37.774930 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:37.774971 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:37.778047 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:37.782751 (+ 4704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:37.782767 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:37.782798 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:37.783604 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:37.784217 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:37.784223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:37.784226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:37.784542 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:37.785363 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:37.785371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:37.785792 (+ 421us) client_negotiation.cc:769] Sending connection context
0812 02:29:37.786096 (+ 304us) client_negotiation.cc:240] Negotiation successful
0812 02:29:37.786596 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":416,"threads_started":1}
I20260812 02:29:37.789876 16543 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40941
I20260812 02:29:37.790424 16543 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:37.791698 16543 heartbeater.cc:511] Master 127.12.8.254:40941 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
I20260812 02:29:37.794014 16140 ts_manager.cc:194] Registered new tserver with Master: 6aef9b40832a48e98a6a195584eed96a (127.12.8.194:37539)
I20260812 02:29:37.795078 16140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:56933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:37.829911 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40941
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:29:38.131757 16379 heartbeater.cc:503] Master 127.12.8.254:40941 was elected leader, sending a full tablet report...
W20260812 02:29:38.253592 16550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:38.253901 16550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:38.253997 16550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:38.261406 16550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:38.261533 16550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:38.261648 16550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:38.271793 16550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:38.273397 16550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:38.274986 16550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:38.291308 16556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:38.291420 16558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:38.291360 16555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:38.292435 16550 server_base.cc:1034] running on GCE node
I20260812 02:29:38.293392 16550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:38.294832 16550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:38.296162 16550 hybrid_clock.cc:648] HybridClock initialized: now 1786501778296050 us; error 61 us; skew 500 ppm
I20260812 02:29:38.296600 16550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501778, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:38.300822 16550 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:29:38.302919 16550 webserver.cc:460] Webserver started at http://127.12.8.195:41093/ using document root <none> and password file <none>
I20260812 02:29:38.303843 16550 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:38.303992 16550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:38.304384 16550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:38.308130 16550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "7326ac9d6c8a420289aa6a19433e87cc"
format_stamp: "Formatted at 2026-08-12 02:29:38 on dist-test-slave-vq2q"
server_key: "a7273fea9c28205988b83d66eb30acdc"
server_key_iv: "a25e3309a415bdf394151d4e28a00d36"
server_key_version: "encryptionkey@0"
I20260812 02:29:38.309110 16550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "7326ac9d6c8a420289aa6a19433e87cc"
format_stamp: "Formatted at 2026-08-12 02:29:38 on dist-test-slave-vq2q"
server_key: "a7273fea9c28205988b83d66eb30acdc"
server_key_iv: "a25e3309a415bdf394151d4e28a00d36"
server_key_version: "encryptionkey@0"
I20260812 02:29:38.316092 16550 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 02:29:38.321251 16565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:38.323272 16550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:29:38.323529 16550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "7326ac9d6c8a420289aa6a19433e87cc"
format_stamp: "Formatted at 2026-08-12 02:29:38 on dist-test-slave-vq2q"
server_key: "a7273fea9c28205988b83d66eb30acdc"
server_key_iv: "a25e3309a415bdf394151d4e28a00d36"
server_key_version: "encryptionkey@0"
I20260812 02:29:38.323807 16550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:38.340088 16550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:38.345538 16550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:38.345919 16550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:38.347157 16550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:38.348879 16550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:38.348971 16550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:38.349114 16550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:38.349202 16550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:38.386935 16550 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:38803
I20260812 02:29:38.387012 16705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:38803 every 8 connection(s)
I20260812 02:29:38.388561 16550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:38.390996 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 16550
I20260812 02:29:38.391207 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:38.391865 12323 external_mini_cluster.cc:1428] Setting key 8d0d15c0b6020a73a292174cc11a86f6
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501778, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:38.414743 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:38.391669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:47439 (local address 127.12.8.254:40941)
0812 02:29:38.393698 (+ 2029us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:38.393704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:38.393730 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:29:38.396322 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:38.396328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:38.396429 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:38.396555 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:38.398944 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.399759 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.401920 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.402370 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.402493 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:38.407056 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:38.407064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:38.407068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:38.407140 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:38.411040 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.412048 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.412053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.412056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.412247 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.412750 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.412754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.412756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.413096 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:38.413305 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:38.414097 (+ 792us) server_negotiation.cc:299] Negotiation successful
0812 02:29:38.414355 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1853,"thread_start_us":1686,"threads_started":1}
I20260812 02:29:38.415683 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:38.392292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40941 (local address 127.12.8.195:47439)
0812 02:29:38.393432 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:38.393506 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:38.395894 (+ 2388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:38.396938 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:38.396950 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:38.397902 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:38.398700 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:38.398724 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.399960 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.399965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:38.401721 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:38.401739 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.402647 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.402653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:38.402935 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:38.403762 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:38.403788 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:38.406626 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:38.411283 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:38.411293 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:38.411309 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:38.411906 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:38.412387 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:38.412391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:38.412393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:38.412630 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:38.413287 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:38.413293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:38.413697 (+ 404us) client_negotiation.cc:769] Sending connection context
0812 02:29:38.413991 (+ 294us) client_negotiation.cc:240] Negotiation successful
0812 02:29:38.414475 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":419,"threads_started":1}
I20260812 02:29:38.417210 16706 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40941
I20260812 02:29:38.417533 16706 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:38.418309 16706 heartbeater.cc:511] Master 127.12.8.254:40941 requested a full tablet report, sending...
I20260812 02:29:38.419975 16140 ts_manager.cc:194] Registered new tserver with Master: 7326ac9d6c8a420289aa6a19433e87cc (127.12.8.195:38803)
I20260812 02:29:38.420986 16140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:47439
I20260812 02:29:38.423391 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501778, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501778, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:38.490500 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:38.474214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.12.8.254:40941)
0812 02:29:38.474487 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:38.474496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:38.475095 (+ 599us) server_negotiation.cc:407] Connection header received
0812 02:29:38.475297 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:38.475309 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:38.475393 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:38.475491 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:38.477153 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.477937 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.479496 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.479974 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.480152 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:38.485030 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:38.485036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:38.485040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:38.485075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:38.487656 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.488419 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.488423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.488425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.488567 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.488943 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.488946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.488948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.489359 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:38.489600 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:38.489906 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 02:29:38.490164 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260812 02:29:38.491271 16139 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42260
I20260812 02:29:38.493942 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16218
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501778, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:38.511940 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:38.495244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.12.8.193:39701)
0812 02:29:38.496019 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:38.496027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:38.496058 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:29:38.496121 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:38.496127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:38.496357 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:38.496494 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:38.498054 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.499099 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.500521 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.501114 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.501272 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:38.505972 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:38.505985 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:38.505999 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:38.506041 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:38.508842 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.509854 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.509862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.509864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.510005 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.510417 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.510419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.510421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.510776 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:38.511045 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:38.511398 (+ 353us) server_negotiation.cc:299] Negotiation successful
0812 02:29:38.511614 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":343,"threads_started":1}
W20260812 02:29:38.513235 16371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34622
W20260812 02:29:38.514389 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:38.514619 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 16218
I20260812 02:29:38.538264 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16388
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501778, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:29:38.556308 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:38.539359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.12.8.194:37539)
0812 02:29:38.540002 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:38.540011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:38.540043 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:29:38.540125 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:38.540133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:38.540411 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:38.540590 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:38.542051 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.543175 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.544517 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.545157 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.545308 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:38.549660 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:38.549678 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:38.549699 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:38.549749 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:38.553153 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.553874 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.553882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.553885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.554053 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.554507 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.554512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.554515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.555002 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:38.555314 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:38.555607 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 02:29:38.555904 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":322,"threads_started":1}
W20260812 02:29:38.557839 16538 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54148
W20260812 02:29:38.559119 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16388: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:38.559316 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 16388
I20260812 02:29:38.582010 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16550
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501778, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:29:38.600827 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:38.583356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.12.8.195:38803)
0812 02:29:38.584171 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:38.584180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:38.584221 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:29:38.584306 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:38.584317 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:38.584632 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:38.584813 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:38.586320 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.587540 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.588836 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:38.589688 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:38.589816 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:38.594365 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:38.594387 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:38.594412 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:38.594462 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:38.597671 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.598458 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.598468 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.598471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.598646 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:38.598971 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:38.598974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:38.598976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:38.599497 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:38.599801 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:38.600088 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 02:29:38.600397 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":437,"threads_started":1}
W20260812 02:29:38.602618 16701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43416
W20260812 02:29:38.604143 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:29:38.604324 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 16550
I20260812 02:29:38.623914 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16093
I20260812 02:29:38.625535 16174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:38.734032 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 16093
2026-08-12T02:29:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4380 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16725](info): set up 2 sockets
Aug 12 02:29:38 dist-test-slave-vq2q krb5kdc[16725](info): commencing operation
krb5kdc: starting...
W20260812 02:29:39.865934 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:40 dist-test-slave-vq2q krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:40.108349 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40415
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:40.466558 16741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:40.466961 16741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:40.467110 16741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:40.474702 16741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:40.474829 16741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:40.474908 16741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:40.474994 16741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:40.475054 16741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:40.485139 16741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:40.486500 16741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:40.487793 16741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:40.499218 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:40.499183 16750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:40.499339 16747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:40.500072 16741 server_base.cc:1034] running on GCE node
I20260812 02:29:40.500859 16741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:40.502236 16741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:40.503459 16741 hybrid_clock.cc:648] HybridClock initialized: now 1786501780503399 us; error 40 us; skew 500 ppm
I20260812 02:29:40.503799 16741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:40 dist-test-slave-vq2q krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501780, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:40.509960 16741 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:40.511497 16741 webserver.cc:460] Webserver started at http://127.12.8.254:36973/ using document root <none> and password file <none>
I20260812 02:29:40.512188 16741 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:40.512305 16741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:40.512610 16741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:40.515611 16741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "d3e8e25225404c66b8b978ba717d48d7"
format_stamp: "Formatted at 2026-08-12 02:29:40 on dist-test-slave-vq2q"
server_key: "61bfb5ab993d7c1153b99d9776381001"
server_key_iv: "e320f4a4473227708109c96d1696bf5f"
server_key_version: "encryptionkey@0"
I20260812 02:29:40.516372 16741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "d3e8e25225404c66b8b978ba717d48d7"
format_stamp: "Formatted at 2026-08-12 02:29:40 on dist-test-slave-vq2q"
server_key: "61bfb5ab993d7c1153b99d9776381001"
server_key_iv: "e320f4a4473227708109c96d1696bf5f"
server_key_version: "encryptionkey@0"
I20260812 02:29:40.521553 16741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 02:29:40.525072 16757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:40.526521 16741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:29:40.526692 16741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "d3e8e25225404c66b8b978ba717d48d7"
format_stamp: "Formatted at 2026-08-12 02:29:40 on dist-test-slave-vq2q"
server_key: "61bfb5ab993d7c1153b99d9776381001"
server_key_iv: "e320f4a4473227708109c96d1696bf5f"
server_key_version: "encryptionkey@0"
I20260812 02:29:40.526894 16741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:40.547768 16741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:40.552373 16741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:40.552650 16741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:40.571292 16741 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:40415
I20260812 02:29:40.571316 16815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:40415 every 8 connection(s)
I20260812 02:29:40.572700 16741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:40.576380 16816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:40.582170 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 16741
I20260812 02:29:40.582376 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:40.583052 12323 external_mini_cluster.cc:1428] Setting key 4b959f81b317563b7993b7bd5c123a2b
I20260812 02:29:40.588095 16816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7: Bootstrap starting.
I20260812 02:29:40.591265 16816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:40.592278 16816 log.cc:824] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:40.595069 16816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7: No bootstrap required, opened a new log
Aug 12 02:29:40 dist-test-slave-vq2q krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:40.603065 16816 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e8e25225404c66b8b978ba717d48d7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40415 } }
I20260812 02:29:40.603459 16816 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:40.603569 16816 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e8e25225404c66b8b978ba717d48d7, State: Initialized, Role: FOLLOWER
I20260812 02:29:40.604322 16816 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e8e25225404c66b8b978ba717d48d7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40415 } }
I20260812 02:29:40.604610 16816 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:40.604753 16816 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:40.604938 16816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:40.605906 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:40.585688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.12.8.254:40415)
0812 02:29:40.586543 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:40.586556 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:40.586597 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:29:40.588353 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:40.588375 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:40.588758 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:40.589114 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:40.590624 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:40.591983 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:40.593227 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:40.593910 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:40.594078 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:40.597901 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:40.597909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:40.597925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:40.597970 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:40.601372 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:40.602067 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:40.602077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:40.602083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:40.602269 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:40.602631 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:40.602635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:40.602637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:40.603180 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:40.603487 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:40.603951 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 02:29:40.604418 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":378,"threads_started":1}
I20260812 02:29:40.607898 16816 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e8e25225404c66b8b978ba717d48d7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40415 } }
I20260812 02:29:40.608505 16816 leader_election.cc:302] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3e8e25225404c66b8b978ba717d48d7; no voters:
I20260812 02:29:40.609275 16816 leader_election.cc:288] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:40.609351 16824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:40.610111 16824 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [term 1 LEADER]: Becoming Leader. State: Replica: d3e8e25225404c66b8b978ba717d48d7, State: Running, Role: LEADER
I20260812 02:29:40.610683 16824 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e8e25225404c66b8b978ba717d48d7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40415 } }
I20260812 02:29:40.611351 16816 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:40.613384 16826 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3e8e25225404c66b8b978ba717d48d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e8e25225404c66b8b978ba717d48d7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40415 } } }
I20260812 02:29:40.613777 16826 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:40.613643 16828 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3e8e25225404c66b8b978ba717d48d7. Latest consensus state: current_term: 1 leader_uuid: "d3e8e25225404c66b8b978ba717d48d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e8e25225404c66b8b978ba717d48d7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40415 } } }
I20260812 02:29:40.613921 16828 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:40.614754 16837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:40.620193 16837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:40.629129 16837 catalog_manager.cc:1347] Generated new cluster ID: 904c30e5c41a4d4688a41bec7edf6b3e
I20260812 02:29:40.629282 16837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:40.654611 16837 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:40.655987 16837 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:40.668471 16837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3e8e25225404c66b8b978ba717d48d7: Generated new TSK 0
I20260812 02:29:40.669566 16837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:40.728561 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40415
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:41.075975 16855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:41.076270 16855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:41.076396 16855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:41.083734 16855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:41.083850 16855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:41.084014 16855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:41.093972 16855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:41.095268 16855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:41.096601 16855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:41.109602 16861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:41.109627 16863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:41.109987 16860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:41.110378 16855 server_base.cc:1034] running on GCE node
I20260812 02:29:41.110985 16855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:41.112000 16855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:41.113260 16855 hybrid_clock.cc:648] HybridClock initialized: now 1786501781113166 us; error 79 us; skew 500 ppm
I20260812 02:29:41.113574 16855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:41 dist-test-slave-vq2q krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:41.116705 16855 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:41.118094 16855 webserver.cc:460] Webserver started at http://127.12.8.193:34793/ using document root <none> and password file <none>
I20260812 02:29:41.118752 16855 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:41.118873 16855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:41.119161 16855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:41.122206 16855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "180111b4012843cf900e08e8319a930f"
format_stamp: "Formatted at 2026-08-12 02:29:41 on dist-test-slave-vq2q"
server_key: "d1b4d5b516c953a12562fb6a2c76c1dd"
server_key_iv: "c90599daf49e8b8d2219905869802808"
server_key_version: "encryptionkey@0"
I20260812 02:29:41.123229 16855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "180111b4012843cf900e08e8319a930f"
format_stamp: "Formatted at 2026-08-12 02:29:41 on dist-test-slave-vq2q"
server_key: "d1b4d5b516c953a12562fb6a2c76c1dd"
server_key_iv: "c90599daf49e8b8d2219905869802808"
server_key_version: "encryptionkey@0"
I20260812 02:29:41.128523 16855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:29:41.132170 16872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:41.133569 16855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:29:41.133729 16855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "180111b4012843cf900e08e8319a930f"
format_stamp: "Formatted at 2026-08-12 02:29:41 on dist-test-slave-vq2q"
server_key: "d1b4d5b516c953a12562fb6a2c76c1dd"
server_key_iv: "c90599daf49e8b8d2219905869802808"
server_key_version: "encryptionkey@0"
I20260812 02:29:41.133932 16855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:41.149230 16855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:41.153450 16855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:41.153731 16855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:41.154829 16855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:41.156211 16855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:41.156292 16855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:41.156424 16855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:41.156489 16855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:41.189260 16855 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:40407
I20260812 02:29:41.189311 17009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:40407 every 8 connection(s)
I20260812 02:29:41.190552 16855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:41.192806 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 16855
I20260812 02:29:41.193001 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:41.193835 12323 external_mini_cluster.cc:1428] Setting key fb9eff9f3ce3798b0f48d140065cebf7
Aug 12 02:29:41 dist-test-slave-vq2q krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:41.211748 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:41.193165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:42013 (local address 127.12.8.254:40415)
0812 02:29:41.193713 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:41.193719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:41.194665 (+ 946us) server_negotiation.cc:407] Connection header received
0812 02:29:41.196551 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:41.196556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:41.196633 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:41.196739 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:41.198204 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:41.199157 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:41.200682 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:41.201281 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:41.201415 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:41.205201 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:41.205209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:41.205213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:41.205254 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:41.208407 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:41.209172 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:41.209201 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:41.209205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:41.209347 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:41.209742 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:41.209746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:41.209749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:41.210153 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:41.210307 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:41.211056 (+ 749us) server_negotiation.cc:299] Negotiation successful
0812 02:29:41.211279 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":253,"threads_started":1}
I20260812 02:29:41.212597 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:41.193699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40415 (local address 127.12.8.193:42013)
0812 02:29:41.194516 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:41.194562 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:41.196297 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:41.196899 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:41.196908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:41.197461 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:41.198035 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:41.198054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:41.199329 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:41.199333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:41.200537 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:41.200549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:41.201501 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:41.201507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:41.201712 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:41.202385 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:41.202412 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:41.204960 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:41.208575 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:41.208583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:41.208595 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:41.209048 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:41.209442 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:41.209445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:41.209447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:41.209641 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:41.210303 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:41.210310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:41.210625 (+ 315us) client_negotiation.cc:769] Sending connection context
0812 02:29:41.210893 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 02:29:41.211326 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":315,"threads_started":1}
I20260812 02:29:41.214391 17010 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40415
I20260812 02:29:41.214726 17010 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:41.215497 17010 heartbeater.cc:511] Master 127.12.8.254:40415 requested a full tablet report, sending...
I20260812 02:29:41.217932 16774 ts_manager.cc:194] Registered new tserver with Master: 180111b4012843cf900e08e8319a930f (127.12.8.193:40407)
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
I20260812 02:29:41.219580 16774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:42013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:41.248780 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40415
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:41.583808 17019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:41.584075 17019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:41.584213 17019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:41.591521 17019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:41.591625 17019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:41.591770 17019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:41.601449 17019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:41.602767 17019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:41.604072 17019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:41.617470 17026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:41.617415 17028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:41.617931 17025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:41.618379 17019 server_base.cc:1034] running on GCE node
I20260812 02:29:41.618943 17019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:41.619839 17019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:41.621037 17019 hybrid_clock.cc:648] HybridClock initialized: now 1786501781620986 us; error 47 us; skew 500 ppm
I20260812 02:29:41.621358 17019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:41 dist-test-slave-vq2q krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:41.624577 17019 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:41.626078 17019 webserver.cc:460] Webserver started at http://127.12.8.194:46877/ using document root <none> and password file <none>
I20260812 02:29:41.626852 17019 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:41.626974 17019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:41.627274 17019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:41.630268 17019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "00659efa913141138450174060eb3e29"
format_stamp: "Formatted at 2026-08-12 02:29:41 on dist-test-slave-vq2q"
server_key: "9370aaf52de112fa5d9bc6c0b1f903c5"
server_key_iv: "e52ae0817183b098e37fd4f7863771cb"
server_key_version: "encryptionkey@0"
I20260812 02:29:41.631039 17019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "00659efa913141138450174060eb3e29"
format_stamp: "Formatted at 2026-08-12 02:29:41 on dist-test-slave-vq2q"
server_key: "9370aaf52de112fa5d9bc6c0b1f903c5"
server_key_iv: "e52ae0817183b098e37fd4f7863771cb"
server_key_version: "encryptionkey@0"
I20260812 02:29:41.636045 17019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:29:41.639554 17039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:41.640832 17019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:29:41.640985 17019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "00659efa913141138450174060eb3e29"
format_stamp: "Formatted at 2026-08-12 02:29:41 on dist-test-slave-vq2q"
server_key: "9370aaf52de112fa5d9bc6c0b1f903c5"
server_key_iv: "e52ae0817183b098e37fd4f7863771cb"
server_key_version: "encryptionkey@0"
I20260812 02:29:41.641206 17019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:41.679342 17019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:41.683545 17019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:41.683842 17019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:41.684875 17019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:41.686272 17019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:41.686352 17019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:41.686483 17019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:41.686539 17019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:41.718549 17019 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:45299
I20260812 02:29:41.718567 17182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:45299 every 8 connection(s)
I20260812 02:29:41.719877 17019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:41.722462 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 17019
I20260812 02:29:41.722661 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:41.723327 12323 external_mini_cluster.cc:1428] Setting key b95a80df07cb38d077b1ecea9bd329ef
Aug 12 02:29:41 dist-test-slave-vq2q krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501781, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:41.746124 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:41.722421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:48309 (local address 127.12.8.254:40415)
0812 02:29:41.729556 (+ 7135us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:41.729560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:41.729578 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:29:41.729629 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:41.729632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:41.729696 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:41.729790 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:41.731851 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:41.732609 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:41.734710 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:41.735373 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:41.735509 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:41.739241 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:41.739247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:41.739252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:41.739293 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:41.742478 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:41.743264 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:41.743270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:41.743273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:41.743440 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:41.743970 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:41.743974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:41.743977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:41.744337 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:41.744484 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:41.745400 (+ 916us) server_negotiation.cc:299] Negotiation successful
0812 02:29:41.745673 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7025,"thread_start_us":6910,"threads_started":1}
I20260812 02:29:41.746932 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:41.722828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40415 (local address 127.12.8.194:48309)
0812 02:29:41.723651 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:41.723698 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:41.725664 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:41.730104 (+ 4440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:41.730116 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:41.730845 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:41.731637 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:41.731664 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:41.732776 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:41.732780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:41.734547 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:41.734562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:41.735588 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:41.735594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:41.735791 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:41.736461 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:41.736483 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:41.738997 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:41.742639 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:41.742646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:41.742660 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:41.743113 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:41.743583 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:41.743587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:41.743590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:41.743816 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:41.744475 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:41.744482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:41.744946 (+ 464us) client_negotiation.cc:769] Sending connection context
0812 02:29:41.745245 (+ 299us) client_negotiation.cc:240] Negotiation successful
0812 02:29:41.745757 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":317,"threads_started":1}
I20260812 02:29:41.748451 17183 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40415
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
I20260812 02:29:41.748771 17183 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:41.749594 17183 heartbeater.cc:511] Master 127.12.8.254:40415 requested a full tablet report, sending...
I20260812 02:29:41.751123 16774 ts_manager.cc:194] Registered new tserver with Master: 00659efa913141138450174060eb3e29 (127.12.8.194:45299)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
I20260812 02:29:41.752158 16774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:48309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:41.780759 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40415
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:42.118129 17190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:42.118396 17190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:42.118515 17190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:42.125896 17190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:42.126011 17190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:42.126163 17190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:42.136106 17190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:42.137466 17190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:42.138870 17190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:42.151409 17201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:42.151437 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:42.151487 17198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:42.152439 17190 server_base.cc:1034] running on GCE node
I20260812 02:29:42.153019 17190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:42.153978 17190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:42.155160 17190 hybrid_clock.cc:648] HybridClock initialized: now 1786501782155099 us; error 53 us; skew 500 ppm
I20260812 02:29:42.155445 17190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501782, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:42.158448 17190 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:29:42.159749 17190 webserver.cc:460] Webserver started at http://127.12.8.195:44699/ using document root <none> and password file <none>
I20260812 02:29:42.160382 17190 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:42.160480 17190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:42.160741 17190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:42.163563 17190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "5e0247b7af8f4c33ade5a2516faddd2f"
format_stamp: "Formatted at 2026-08-12 02:29:42 on dist-test-slave-vq2q"
server_key: "44ecf6f9deb89361ca25b2d5649518fc"
server_key_iv: "74484eae2d0326babfcce57284e96ee4"
server_key_version: "encryptionkey@0"
I20260812 02:29:42.164291 17190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "5e0247b7af8f4c33ade5a2516faddd2f"
format_stamp: "Formatted at 2026-08-12 02:29:42 on dist-test-slave-vq2q"
server_key: "44ecf6f9deb89361ca25b2d5649518fc"
server_key_iv: "74484eae2d0326babfcce57284e96ee4"
server_key_version: "encryptionkey@0"
I20260812 02:29:42.169346 17190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 02:29:42.172807 17211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:42.174234 17190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:29:42.174383 17190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "5e0247b7af8f4c33ade5a2516faddd2f"
format_stamp: "Formatted at 2026-08-12 02:29:42 on dist-test-slave-vq2q"
server_key: "44ecf6f9deb89361ca25b2d5649518fc"
server_key_iv: "74484eae2d0326babfcce57284e96ee4"
server_key_version: "encryptionkey@0"
I20260812 02:29:42.174595 17190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:42.196535 17190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:42.200390 17190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:42.200660 17190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:42.201704 17190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:42.203053 17190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:42.203130 17190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:42.203244 17190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:42.203294 17190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:42.222695 17010 heartbeater.cc:503] Master 127.12.8.254:40415 was elected leader, sending a full tablet report...
I20260812 02:29:42.233594 17190 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:43153
I20260812 02:29:42.233635 17350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:43153 every 8 connection(s)
I20260812 02:29:42.234987 17190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:42.245101 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 17190
I20260812 02:29:42.245306 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:42.245915 12323 external_mini_cluster.cc:1428] Setting key 6ec6dcd3f492b94be00f98ff4ebf32d6
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501782, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:42.255335 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:42.237933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:42067 (local address 127.12.8.254:40415)
0812 02:29:42.238165 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:42.238172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:42.239392 (+ 1220us) server_negotiation.cc:407] Connection header received
0812 02:29:42.241234 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:42.241238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:42.241330 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:42.241425 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:42.242950 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.243702 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.245405 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.246041 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.246185 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:42.250061 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:42.250066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:42.250070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:42.250107 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:42.252538 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.253279 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.253283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.253286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.253398 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.253804 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.253808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.253810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.254127 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:42.254249 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:42.254877 (+ 628us) server_negotiation.cc:299] Negotiation successful
0812 02:29:42.255083 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:29:42.256325 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:42.238386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40415 (local address 127.12.8.195:42067)
0812 02:29:42.239214 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:42.239258 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:42.240968 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:42.241635 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:42.241643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:42.242194 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:42.242758 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:42.242775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.243846 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.243849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:42.245240 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:42.245257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.246256 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.246262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:42.246474 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:42.247162 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:42.247193 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:42.249862 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:42.252672 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:42.252679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:42.252691 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:42.253146 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:42.253494 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:42.253497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:42.253500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:42.253689 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:42.254221 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:42.254227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:42.254567 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 02:29:42.254791 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 02:29:42.255195 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":333,"threads_started":1}
I20260812 02:29:42.257716 17351 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40415
I20260812 02:29:42.258037 17351 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:42.258777 17351 heartbeater.cc:511] Master 127.12.8.254:40415 requested a full tablet report, sending...
I20260812 02:29:42.260468 16773 ts_manager.cc:194] Registered new tserver with Master: 5e0247b7af8f4c33ade5a2516faddd2f (127.12.8.195:43153)
I20260812 02:29:42.261569 16773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:42067
I20260812 02:29:42.261631 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:42.322232 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:42.305018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.12.8.193:40407)
0812 02:29:42.305693 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:42.305700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:42.305744 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:29:42.305810 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:42.305817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:42.306031 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:42.306157 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:42.307530 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.308574 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.309823 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.310403 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.310515 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:42.314931 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:42.314983 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:42.315005 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:42.315074 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:42.318688 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.319611 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.319621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.319623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.319814 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.320206 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.320210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.320212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.320717 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:42.321044 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:42.321431 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 02:29:42.321793 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":341,"threads_started":1}
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:42.342348 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:42.327939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.12.8.254:40415)
0812 02:29:42.328240 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:42.328247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:42.328395 (+ 148us) server_negotiation.cc:407] Connection header received
0812 02:29:42.328553 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:42.328558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:42.328670 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:42.328773 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:42.330185 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.330998 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.332270 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.332695 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.332810 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:42.336948 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:42.336954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:42.336958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:42.336996 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:42.339633 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.340288 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.340292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.340295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.340435 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.340787 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.340791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.340793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.341149 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:42.341446 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:42.341770 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 02:29:42.342019 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260812 02:29:42.343128 16773 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42240
I20260812 02:29:42.344437 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16855
I20260812 02:29:42.359103 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:42.345501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.12.8.193:40407)
0812 02:29:42.345697 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:42.345703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:42.346117 (+ 414us) server_negotiation.cc:407] Connection header received
0812 02:29:42.346296 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:42.346301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:42.346393 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:42.346503 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:42.348006 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.348718 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.350215 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.350677 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.350791 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:42.353636 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:42.353641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:42.353645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:42.353683 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:42.356370 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.357131 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.357135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.357138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.357327 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.357718 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.357721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.357723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.358002 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:42.358196 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:42.358638 (+ 442us) server_negotiation.cc:299] Negotiation successful
0812 02:29:42.358831 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:29:42.360005 17005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:42.453151 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 16855
I20260812 02:29:42.476730 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 17019
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:29:42.494740 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:42.477978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.12.8.194:45299)
0812 02:29:42.478657 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:42.478667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:42.478697 (+ 30us) server_negotiation.cc:407] Connection header received
0812 02:29:42.478766 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:42.478772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:42.479009 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:42.479158 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:42.480539 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.481753 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.483117 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.483650 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.483750 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:42.488461 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:42.488473 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:42.488488 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:42.488529 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:42.491742 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.492600 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.492608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.492610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.492761 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.493141 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.493144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.493146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.493565 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:42.493812 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:42.494119 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 02:29:42.494334 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":319,"threads_started":1}
I20260812 02:29:42.496281 17177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:42.582592 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 17019
I20260812 02:29:42.604764 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 17190
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:29:42.624665 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:42.606205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.12.8.195:43153)
0812 02:29:42.607117 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:42.607126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:42.607165 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:29:42.607249 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:42.607257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:42.607575 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:42.607746 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:42.609425 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.610741 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.612297 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:42.613074 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:42.613252 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:42.618139 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:42.618162 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:42.618183 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:42.618250 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:42.621684 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.622456 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.622465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.622467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.622657 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:42.623076 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:42.623079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:42.623080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:42.623506 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:42.623808 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:42.624034 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 02:29:42.624306 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":483,"threads_started":1}
I20260812 02:29:42.626523 17346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:42.713526 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 17190
I20260812 02:29:42.735703 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 16741
I20260812 02:29:42.737085 16811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:42.874266 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 16741
2026-08-12T02:29:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4139 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[17374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[17374](info): set up 2 sockets
Aug 12 02:29:42 dist-test-slave-vq2q krb5kdc[17374](info): commencing operation
krb5kdc: starting...
W20260812 02:29:44.023260 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.006s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:44 dist-test-slave-vq2q krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:44.317138 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45885
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:45885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:44.663897 17389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:44.664228 17389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:44.664330 17389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:44.671887 17389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:44.672026 17389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:44.672075 17389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:44.672134 17389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:44.672214 17389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:44.682534 17389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:45885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:44.684002 17389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:44.685397 17389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:44.698168 17394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:44.698297 17396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:44.698238 17399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:44.700054 17389 server_base.cc:1034] running on GCE node
I20260812 02:29:44.700938 17389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:44.702528 17389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:44.703761 17389 hybrid_clock.cc:648] HybridClock initialized: now 1786501784703699 us; error 60 us; skew 500 ppm
I20260812 02:29:44.704197 17389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:44 dist-test-slave-vq2q krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501784, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:44.707818 17389 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:44.709257 17389 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:44.709372 17389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:44.709789 17389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:44.713244 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "e38838c15b494452b63095a4d5887dc7"
format_stamp: "Formatted at 2026-08-12 02:29:44 on dist-test-slave-vq2q"
server_key: "d9a859da4feae2a38bd0f737c34873ef"
server_key_iv: "268dbf25d5930a933cc51f9463a3b5fc"
server_key_version: "encryptionkey@0"
I20260812 02:29:44.714129 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "e38838c15b494452b63095a4d5887dc7"
format_stamp: "Formatted at 2026-08-12 02:29:44 on dist-test-slave-vq2q"
server_key: "d9a859da4feae2a38bd0f737c34873ef"
server_key_iv: "268dbf25d5930a933cc51f9463a3b5fc"
server_key_version: "encryptionkey@0"
I20260812 02:29:44.719931 17389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 02:29:44.724336 17407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:44.726318 17389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 02:29:44.726531 17389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "e38838c15b494452b63095a4d5887dc7"
format_stamp: "Formatted at 2026-08-12 02:29:44 on dist-test-slave-vq2q"
server_key: "d9a859da4feae2a38bd0f737c34873ef"
server_key_iv: "268dbf25d5930a933cc51f9463a3b5fc"
server_key_version: "encryptionkey@0"
I20260812 02:29:44.726774 17389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:44.746935 17389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:44.751478 17389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:44.751750 17389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:44.773069 17389 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:45885
I20260812 02:29:44.773134 17469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:45885 every 8 connection(s)
I20260812 02:29:44.774149 17389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:44.778487 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 17389
I20260812 02:29:44.778685 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:44.779332 17470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:44.779363 12323 external_mini_cluster.cc:1428] Setting key f38273f065c0c889a1fadd1de96259c5
Aug 12 02:29:44 dist-test-slave-vq2q krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:44.798230 17470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7: Bootstrap starting.
I20260812 02:29:44.804422 17470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:44.806439 17470 log.cc:824] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:44.811342 17470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7: No bootstrap required, opened a new log
I20260812 02:29:44.811385 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:44.781895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.12.8.254:45885)
0812 02:29:44.783005 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:44.783034 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:44.783106 (+ 72us) server_negotiation.cc:407] Connection header received
0812 02:29:44.785745 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:44.785789 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:44.786704 (+ 915us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:44.787447 (+ 743us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:44.789068 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:44.791353 (+ 2285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:44.793143 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:44.794487 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:44.794749 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:44.799509 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:44.799524 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:44.799563 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:44.799634 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:44.804801 (+ 5167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:44.805923 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:44.805936 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:44.805949 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:44.806368 (+ 419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:44.807054 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:44.807060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:44.807064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:44.808034 (+ 970us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:44.808383 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:44.809002 (+ 619us) server_negotiation.cc:299] Negotiation successful
0812 02:29:44.809650 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":446,"threads_started":1}
I20260812 02:29:44.821713 17470 raft_consensus.cc:348] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38838c15b494452b63095a4d5887dc7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45885 } }
I20260812 02:29:44.822280 17470 raft_consensus.cc:374] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:44.822377 17470 raft_consensus.cc:729] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38838c15b494452b63095a4d5887dc7, State: Initialized, Role: FOLLOWER
I20260812 02:29:44.823004 17470 consensus_queue.cc:260] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38838c15b494452b63095a4d5887dc7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45885 } }
I20260812 02:29:44.823249 17470 raft_consensus.cc:388] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:44.823395 17470 raft_consensus.cc:482] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:44.823550 17470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:44.826409 17470 raft_consensus.cc:504] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38838c15b494452b63095a4d5887dc7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45885 } }
I20260812 02:29:44.826975 17470 leader_election.cc:302] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e38838c15b494452b63095a4d5887dc7; no voters:
I20260812 02:29:44.827703 17470 leader_election.cc:288] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:44.827987 17477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:44.828985 17477 raft_consensus.cc:686] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [term 1 LEADER]: Becoming Leader. State: Replica: e38838c15b494452b63095a4d5887dc7, State: Running, Role: LEADER
I20260812 02:29:44.829802 17477 consensus_queue.cc:237] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38838c15b494452b63095a4d5887dc7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45885 } }
I20260812 02:29:44.830202 17470 sys_catalog.cc:564] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:44.834475 17479 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38838c15b494452b63095a4d5887dc7. Latest consensus state: current_term: 1 leader_uuid: "e38838c15b494452b63095a4d5887dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38838c15b494452b63095a4d5887dc7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45885 } } }
I20260812 02:29:44.834405 17478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e38838c15b494452b63095a4d5887dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38838c15b494452b63095a4d5887dc7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45885 } } }
I20260812 02:29:44.835028 17479 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:44.835095 17478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:44.836057 17487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:44.843284 17487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:44.854120 17487 catalog_manager.cc:1347] Generated new cluster ID: 593dea7dc1d740f581741a84baf22748
I20260812 02:29:44.854269 17487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:44.874943 17487 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:44.877333 17487 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:44.888343 17487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7: Generated new TSK 0
I20260812 02:29:44.889400 17487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:44.981573 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45885
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:45.349399 17500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:45.349713 17500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:45.349815 17500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:45.357309 17500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:45.357431 17500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:45.357553 17500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:45.367710 17500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:45.369062 17500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:45.370404 17500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:45.384833 17508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:45.384897 17505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:45.384914 17506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:45.386416 17500 server_base.cc:1034] running on GCE node
I20260812 02:29:45.387303 17500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:45.388661 17500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:45.389890 17500 hybrid_clock.cc:648] HybridClock initialized: now 1786501785389829 us; error 50 us; skew 500 ppm
I20260812 02:29:45.390283 17500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:45 dist-test-slave-vq2q krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:45.394243 17500 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:45.395731 17500 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:45.395851 17500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:45.396224 17500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:45.400014 17500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "b48418d221d64271985dd05d0ac39705"
format_stamp: "Formatted at 2026-08-12 02:29:45 on dist-test-slave-vq2q"
server_key: "d670cebef3527d2cec2b635451b7303e"
server_key_iv: "0895c302da859e2f747d7842d544b175"
server_key_version: "encryptionkey@0"
I20260812 02:29:45.401055 17500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "b48418d221d64271985dd05d0ac39705"
format_stamp: "Formatted at 2026-08-12 02:29:45 on dist-test-slave-vq2q"
server_key: "d670cebef3527d2cec2b635451b7303e"
server_key_iv: "0895c302da859e2f747d7842d544b175"
server_key_version: "encryptionkey@0"
I20260812 02:29:45.407590 17500 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 02:29:45.412498 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:45.414590 17500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:29:45.414873 17500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "b48418d221d64271985dd05d0ac39705"
format_stamp: "Formatted at 2026-08-12 02:29:45 on dist-test-slave-vq2q"
server_key: "d670cebef3527d2cec2b635451b7303e"
server_key_iv: "0895c302da859e2f747d7842d544b175"
server_key_version: "encryptionkey@0"
I20260812 02:29:45.415174 17500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:45.450466 17500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:45.456278 17500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:45.456630 17500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:45.458026 17500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:45.459862 17500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:45.459983 17500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:45.460105 17500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:45.460186 17500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:45.509131 17500 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:40603
I20260812 02:29:45.509325 17652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:40603 every 8 connection(s)
I20260812 02:29:45.510440 17500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:45.517433 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 17500
I20260812 02:29:45.517664 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:45.518532 12323 external_mini_cluster.cc:1428] Setting key fc5ae494d9785706c601497e7b9d1a14
Aug 12 02:29:45 dist-test-slave-vq2q krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:45.541431 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:45.513515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:54623 (local address 127.12.8.254:45885)
0812 02:29:45.514018 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:45.514024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:45.515281 (+ 1257us) server_negotiation.cc:407] Connection header received
0812 02:29:45.517882 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:45.517888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:45.517985 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:45.518095 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:45.520602 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:45.521521 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:45.523895 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:45.524449 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:45.524566 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:45.533632 (+ 9066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:45.533640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:45.533645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:45.533693 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:45.536790 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:45.538254 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:45.538261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:45.538265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:45.538443 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:45.539030 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:45.539036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:45.539040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:45.539454 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:45.539626 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:45.540805 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0812 02:29:45.541060 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":265,"threads_started":1}
I20260812 02:29:45.543087 17656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:45.514166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45885 (local address 127.12.8.193:54623)
0812 02:29:45.515087 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:45.515144 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:45.517386 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:45.518585 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:45.518598 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:45.519466 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:45.520401 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:45.520430 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:45.521687 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:45.521692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:45.523691 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:45.523709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:45.524863 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:45.524869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:45.525125 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:45.530419 (+ 5294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:45.530461 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:45.533313 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:45.537070 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:45.537086 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:45.537114 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:45.538066 (+ 952us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:45.538593 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:45.538598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:45.538601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:45.538890 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:45.539613 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:45.539630 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:45.540221 (+ 591us) client_negotiation.cc:769] Sending connection context
0812 02:29:45.540597 (+ 376us) client_negotiation.cc:240] Negotiation successful
0812 02:29:45.541379 (+ 782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":355,"threads_started":1}
I20260812 02:29:45.545217 17653 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45885
I20260812 02:29:45.545578 17653 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:45.546350 17653 heartbeater.cc:511] Master 127.12.8.254:45885 requested a full tablet report, sending...
I20260812 02:29:45.549113 17428 ts_manager.cc:194] Registered new tserver with Master: b48418d221d64271985dd05d0ac39705 (127.12.8.193:40603)
I20260812 02:29:45.551659 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:54623
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:45.589972 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45885
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:45.942866 17659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:45.943187 17659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:45.943303 17659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:45.951033 17659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:45.951160 17659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:45.951318 17659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:45.962152 17659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:45.963635 17659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:45.965046 17659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:45.979116 17664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:45.979161 17667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:45.978996 17665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:45.980489 17659 server_base.cc:1034] running on GCE node
I20260812 02:29:45.981153 17659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:45.982313 17659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:45.983561 17659 hybrid_clock.cc:648] HybridClock initialized: now 1786501785983498 us; error 81 us; skew 500 ppm
I20260812 02:29:45.983916 17659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:45 dist-test-slave-vq2q krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:45.987351 17659 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:45.988641 17659 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:45.988749 17659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:45.989135 17659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:45.992373 17659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "19f08323ab57425497b3a8f9ec1c55d1"
format_stamp: "Formatted at 2026-08-12 02:29:45 on dist-test-slave-vq2q"
server_key: "1ec37fae15b2f19567557d501fbfd7c3"
server_key_iv: "ec549400f4d7c4094f6c454dcb261e71"
server_key_version: "encryptionkey@0"
I20260812 02:29:45.993255 17659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "19f08323ab57425497b3a8f9ec1c55d1"
format_stamp: "Formatted at 2026-08-12 02:29:45 on dist-test-slave-vq2q"
server_key: "1ec37fae15b2f19567557d501fbfd7c3"
server_key_iv: "ec549400f4d7c4094f6c454dcb261e71"
server_key_version: "encryptionkey@0"
I20260812 02:29:45.998911 17659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:29:46.002950 17674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:46.004655 17659 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:46.004881 17659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "19f08323ab57425497b3a8f9ec1c55d1"
format_stamp: "Formatted at 2026-08-12 02:29:45 on dist-test-slave-vq2q"
server_key: "1ec37fae15b2f19567557d501fbfd7c3"
server_key_iv: "ec549400f4d7c4094f6c454dcb261e71"
server_key_version: "encryptionkey@0"
I20260812 02:29:46.005112 17659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:46.021250 17659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:46.025964 17659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:46.026243 17659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:46.027345 17659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:46.029038 17659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:46.029157 17659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:46.029302 17659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:46.029385 17659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:46.066582 17659 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:37755
I20260812 02:29:46.066630 17805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:37755 every 8 connection(s)
I20260812 02:29:46.067477 17659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:46.071112 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 17659
I20260812 02:29:46.071316 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:46.071961 12323 external_mini_cluster.cc:1428] Setting key 34e955843f98dbbf4d7f577a3595fde9
Aug 12 02:29:46 dist-test-slave-vq2q krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:46.094621 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.070523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:51175 (local address 127.12.8.254:45885)
0812 02:29:46.071147 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.071155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.072410 (+ 1255us) server_negotiation.cc:407] Connection header received
0812 02:29:46.074929 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.074933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.075001 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.075110 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:46.077743 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.078477 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.080890 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.081445 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.081560 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.086677 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:46.086687 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:46.086692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:46.086753 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:46.090117 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:46.091356 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:46.091362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:46.091365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:46.091547 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:46.092227 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:46.092231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:46.092234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:46.092590 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:46.092730 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.093982 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.094288 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":314,"threads_started":1}
I20260812 02:29:46.095974 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.071043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45885 (local address 127.12.8.194:51175)
0812 02:29:46.072164 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:46.072242 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:46.074549 (+ 2307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:46.075468 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:46.075483 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:46.076363 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:46.077480 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.077512 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.078669 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.078675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.080684 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.080702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.081744 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.081751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.082068 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.083005 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:46.083045 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:46.086270 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:46.090379 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:46.090391 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:46.090412 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:46.091153 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:46.091692 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:46.091698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:46.091702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:46.092065 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:46.092735 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:46.092744 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:46.093329 (+ 585us) client_negotiation.cc:769] Sending connection context
0812 02:29:46.093756 (+ 427us) client_negotiation.cc:240] Negotiation successful
0812 02:29:46.094472 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":461,"threads_started":1}
I20260812 02:29:46.098169 17806 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45885
I20260812 02:29:46.098604 17806 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:46.099511 17806 heartbeater.cc:511] Master 127.12.8.254:45885 requested a full tablet report, sending...
I20260812 02:29:46.101742 17428 ts_manager.cc:194] Registered new tserver with Master: 19f08323ab57425497b3a8f9ec1c55d1 (127.12.8.194:37755)
I20260812 02:29:46.102742 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:51175
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:46.140797 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45885
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:46.488340 17812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:46.488637 17812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:46.488768 17812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:46.496104 17812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:46.496233 17812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:46.496405 17812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:46.506633 17812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:46.508350 17812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:46.510088 17812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:46.525979 17818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:46.526003 17817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:46.526464 17820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:46.527540 17812 server_base.cc:1034] running on GCE node
I20260812 02:29:46.528206 17812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:46.529306 17812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:46.530596 17812 hybrid_clock.cc:648] HybridClock initialized: now 1786501786530512 us; error 84 us; skew 500 ppm
I20260812 02:29:46.530936 17812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:46 dist-test-slave-vq2q krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501786, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:46.534334 17812 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:29:46.535660 17812 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:46.535761 17812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:46.536070 17812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:46.539043 17812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "f5d1e341ccca4107afdfa11ae04719de"
format_stamp: "Formatted at 2026-08-12 02:29:46 on dist-test-slave-vq2q"
server_key: "d1deafa7b3f15c94fa0e9817a68fed34"
server_key_iv: "2f8498a86b264b7db8be43baa906e834"
server_key_version: "encryptionkey@0"
I20260812 02:29:46.539835 17812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "f5d1e341ccca4107afdfa11ae04719de"
format_stamp: "Formatted at 2026-08-12 02:29:46 on dist-test-slave-vq2q"
server_key: "d1deafa7b3f15c94fa0e9817a68fed34"
server_key_iv: "2f8498a86b264b7db8be43baa906e834"
server_key_version: "encryptionkey@0"
I20260812 02:29:46.545403 17812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 02:29:46.549360 17829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:46.551023 17812 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:29:46.551201 17812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "f5d1e341ccca4107afdfa11ae04719de"
format_stamp: "Formatted at 2026-08-12 02:29:46 on dist-test-slave-vq2q"
server_key: "d1deafa7b3f15c94fa0e9817a68fed34"
server_key_iv: "2f8498a86b264b7db8be43baa906e834"
server_key_version: "encryptionkey@0"
I20260812 02:29:46.551445 17812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:46.555467 17653 heartbeater.cc:503] Master 127.12.8.254:45885 was elected leader, sending a full tablet report...
I20260812 02:29:46.592906 17812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:46.597321 17812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:46.597579 17812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:46.598649 17812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:46.600133 17812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:46.600236 17812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:46.600353 17812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:46.600431 17812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:46.637171 17812 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:43097
I20260812 02:29:46.637351 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:43097 every 8 connection(s)
I20260812 02:29:46.638079 17812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:46.642308 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 17812
I20260812 02:29:46.642566 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:46.643241 12323 external_mini_cluster.cc:1428] Setting key fbf4858d99db76bed024b23d8ca5c71e
Aug 12 02:29:46 dist-test-slave-vq2q krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501786, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:46.659996 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.641212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:44515 (local address 127.12.8.254:45885)
0812 02:29:46.641763 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.641770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.643127 (+ 1357us) server_negotiation.cc:407] Connection header received
0812 02:29:46.645456 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.645462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.645553 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.645657 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:46.647342 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.648129 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.649765 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.650183 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.650293 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.654088 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:46.654094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:46.654097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:46.654137 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:46.656791 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:46.657591 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:46.657595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:46.657598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:46.657726 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:46.658151 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:46.658154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:46.658156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:46.658453 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:46.658602 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.659324 (+ 722us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.659568 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
I20260812 02:29:46.660941 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.641766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45885 (local address 127.12.8.195:44515)
0812 02:29:46.642924 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:46.642987 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:46.645115 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:46.645959 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:46.645968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:46.646541 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:46.647166 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.647185 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.648277 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.648281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.649611 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.649623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.650419 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.650427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.650645 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.651386 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:46.651410 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:46.653859 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:46.656941 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:46.656950 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:46.656967 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:46.657464 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:46.657840 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:46.657844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:46.657848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:46.658034 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:46.658588 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:46.658595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:46.658946 (+ 351us) client_negotiation.cc:769] Sending connection context
0812 02:29:46.659190 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 02:29:46.659729 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":529,"threads_started":1}
I20260812 02:29:46.662532 17964 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45885
I20260812 02:29:46.662976 17964 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:46.663848 17964 heartbeater.cc:511] Master 127.12.8.254:45885 requested a full tablet report, sending...
I20260812 02:29:46.665709 17428 ts_manager.cc:194] Registered new tserver with Master: f5d1e341ccca4107afdfa11ae04719de (127.12.8.195:43097)
I20260812 02:29:46.666579 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:44515
I20260812 02:29:46.672643 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:29:46.694233 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.678222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.12.8.254:45885)
0812 02:29:46.678410 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.678415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.679020 (+ 605us) server_negotiation.cc:407] Connection header received
0812 02:29:46.679263 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.679268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.679344 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.679467 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:46.681168 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.682014 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.683615 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.684117 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.684237 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.687433 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:46.687438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:46.687441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:46.687479 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:46.690356 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:46.691506 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:46.691510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:46.691513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:46.691656 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:46.692272 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:46.692277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:46.692279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:46.692646 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:46.692891 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.693698 (+ 807us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.693976 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:29:46.699903 17428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:46.707111 17428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:46.743616 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.722025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.12.8.193:40603)
0812 02:29:46.734302 (+ 12277us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.734312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.734344 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:29:46.734644 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.734653 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.734944 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.735164 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:46.736100 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.737815 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.740564 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.742105 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.742249 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.742552 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.742965 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.743207 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12082,"thread_start_us":12349,"threads_started":1}
I20260812 02:29:46.743628 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.721902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:40603 (local address 127.0.0.1:38440)
0812 02:29:46.734250 (+ 12348us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:46.734271 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:46.734396 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:46.735356 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:46.735363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:46.735397 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:46.735863 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.735877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.738053 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.738059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.740342 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.740362 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.742343 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.742350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.742623 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.742684 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 02:29:46.742867 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 02:29:46.743135 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12209,"thread_start_us":12821,"threads_started":1}
I20260812 02:29:46.748163 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.721849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:43097 (local address 127.0.0.1:52082)
0812 02:29:46.732496 (+ 10647us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:46.732555 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:46.732774 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:46.739820 (+ 7046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:46.739827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:46.739868 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:46.740560 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.740584 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.742641 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.742647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.744900 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.744915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.747234 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.747244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.747459 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.747500 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 02:29:46.747604 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 02:29:46.747777 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10305,"thread_start_us":12133,"threads_started":1}
I20260812 02:29:46.748983 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.722668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.12.8.195:43097)
0812 02:29:46.738835 (+ 16167us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.738849 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.738904 (+ 55us) server_negotiation.cc:407] Connection header received
0812 02:29:46.739058 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.739069 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.739442 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.740612 (+ 1170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:46.740968 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.742431 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.745130 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.746665 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.746802 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.747060 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.748472 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.748656 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15883,"thread_start_us":15633,"threads_started":1}
I20260812 02:29:46.748383 17579 tablet_service.cc:1467] Processing CreateTablet for tablet 34831398682543e3941566ca0f4c5532 (DEFAULT_TABLE table=test-table [id=f93a2da9504f4d2ba4202484afe0700c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:46.749302 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.722202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:37755 (local address 127.0.0.1:55988)
0812 02:29:46.735052 (+ 12850us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:46.735069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:46.735175 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:46.741076 (+ 5901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:46.741081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:46.741109 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:46.741519 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.741532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.744641 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.744646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.746201 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.746215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.748520 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.748527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.748699 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.748727 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:29:46.748789 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 02:29:46.748914 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12725,"thread_start_us":10799,"threads_started":1}
I20260812 02:29:46.750216 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34831398682543e3941566ca0f4c5532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:46.750313 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.722854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.12.8.194:37755)
0812 02:29:46.740417 (+ 17563us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.740425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.740455 (+ 30us) server_negotiation.cc:407] Connection header received
0812 02:29:46.740569 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.740576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.740844 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.741060 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:46.741718 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.744477 (+ 2759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.746419 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.747961 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.748116 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.749716 (+ 1600us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.749805 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.749969 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17376,"thread_start_us":17248,"threads_started":1}
I20260812 02:29:46.753881 17886 tablet_service.cc:1467] Processing CreateTablet for tablet 34831398682543e3941566ca0f4c5532 (DEFAULT_TABLE table=test-table [id=f93a2da9504f4d2ba4202484afe0700c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:46.754778 17733 tablet_service.cc:1467] Processing CreateTablet for tablet 34831398682543e3941566ca0f4c5532 (DEFAULT_TABLE table=test-table [id=f93a2da9504f4d2ba4202484afe0700c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:46.755653 17886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34831398682543e3941566ca0f4c5532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:46.756439 17733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34831398682543e3941566ca0f4c5532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:46.765944 17985 tablet_bootstrap.cc:492] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Bootstrap starting.
I20260812 02:29:46.771325 17986 tablet_bootstrap.cc:492] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Bootstrap starting.
I20260812 02:29:46.772046 17985 tablet_bootstrap.cc:654] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:46.773931 17985 log.cc:824] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:46.778532 17986 tablet_bootstrap.cc:654] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:46.780177 17986 log.cc:824] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:46.781649 17987 tablet_bootstrap.cc:492] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Bootstrap starting.
I20260812 02:29:46.785924 17987 tablet_bootstrap.cc:654] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:46.785988 17985 tablet_bootstrap.cc:492] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: No bootstrap required, opened a new log
I20260812 02:29:46.786546 17985 ts_tablet_manager.cc:1397] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20260812 02:29:46.789220 17987 log.cc:824] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:46.792936 17987 tablet_bootstrap.cc:492] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: No bootstrap required, opened a new log
I20260812 02:29:46.793171 17986 tablet_bootstrap.cc:492] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: No bootstrap required, opened a new log
I20260812 02:29:46.793277 17987 ts_tablet_manager.cc:1397] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260812 02:29:46.793550 17986 ts_tablet_manager.cc:1397] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20260812 02:29:46.798882 17985 raft_consensus.cc:348] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.799379 17985 raft_consensus.cc:374] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:46.799479 17985 raft_consensus.cc:729] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48418d221d64271985dd05d0ac39705, State: Initialized, Role: FOLLOWER
I20260812 02:29:46.800148 17985 consensus_queue.cc:260] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.802983 17986 raft_consensus.cc:348] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.803436 17986 raft_consensus.cc:374] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:46.803561 17986 raft_consensus.cc:729] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d1e341ccca4107afdfa11ae04719de, State: Initialized, Role: FOLLOWER
I20260812 02:29:46.803943 17987 raft_consensus.cc:348] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.804401 17987 raft_consensus.cc:374] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:46.804491 17987 raft_consensus.cc:729] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f08323ab57425497b3a8f9ec1c55d1, State: Initialized, Role: FOLLOWER
I20260812 02:29:46.804476 17986 consensus_queue.cc:260] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.805169 17985 ts_tablet_manager.cc:1428] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260812 02:29:46.805152 17987 consensus_queue.cc:260] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.806341 17964 heartbeater.cc:503] Master 127.12.8.254:45885 was elected leader, sending a full tablet report...
I20260812 02:29:46.806803 17806 heartbeater.cc:503] Master 127.12.8.254:45885 was elected leader, sending a full tablet report...
I20260812 02:29:46.807430 17987 ts_tablet_manager.cc:1428] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260812 02:29:46.807957 17986 ts_tablet_manager.cc:1428] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Time spent starting tablet: real 0.014s user 0.002s sys 0.011s
W20260812 02:29:46.819450 17807 tablet.cc:2369] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:29:46.889729 17965 tablet.cc:2369] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:46.943245 17992 raft_consensus.cc:482] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:46.943562 17992 raft_consensus.cc:504] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.945412 17992 leader_election.cc:288] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5d1e341ccca4107afdfa11ae04719de (127.12.8.195:43097), 19f08323ab57425497b3a8f9ec1c55d1 (127.12.8.194:37755)
I20260812 02:29:46.952361 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.946015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:42051 (local address 127.12.8.195:43097)
0812 02:29:46.946291 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.946299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.946862 (+ 563us) server_negotiation.cc:407] Connection header received
0812 02:29:46.947069 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.947073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.947145 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.947240 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:46.947815 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.948569 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.950640 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.951442 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.951603 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.951683 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.951969 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.952105 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 02:29:46.952581 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.945886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:43097 (local address 127.12.8.193:42051)
0812 02:29:46.946713 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:46.946729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:46.946916 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:46.947270 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:46.947274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:46.947304 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:46.947662 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.947672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.948711 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.948715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.950465 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.950479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.951652 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.951658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.951815 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.951842 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:29:46.951931 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 02:29:46.952084 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":1,"thread_start_us":816,"threads_started":1}
I20260812 02:29:46.953276 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.946079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:36019 (local address 127.12.8.194:37755)
0812 02:29:46.946312 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.946318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.947190 (+ 872us) server_negotiation.cc:407] Connection header received
0812 02:29:46.947363 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.947368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.947443 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.947524 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:29:46.948189 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.948988 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.950976 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.951837 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.952003 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.952102 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.952706 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.952867 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:29:46.953892 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.945905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:37755 (local address 127.12.8.193:36019)
0812 02:29:46.946992 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:46.947015 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:46.947203 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:46.947567 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:46.947571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:46.947616 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:29:46.948010 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.948020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.949142 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.949146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.950805 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:46.950819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.952077 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.952083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:46.952333 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.952362 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:29:46.953382 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0812 02:29:46.953554 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":47,"thread_start_us":412,"threads_started":1}
I20260812 02:29:46.953882 17908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34831398682543e3941566ca0f4c5532" candidate_uuid: "b48418d221d64271985dd05d0ac39705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d1e341ccca4107afdfa11ae04719de" is_pre_election: true
I20260812 02:29:46.954468 17908 raft_consensus.cc:2454] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b48418d221d64271985dd05d0ac39705 in term 0.
I20260812 02:29:46.955080 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34831398682543e3941566ca0f4c5532" candidate_uuid: "b48418d221d64271985dd05d0ac39705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f08323ab57425497b3a8f9ec1c55d1" is_pre_election: true
I20260812 02:29:46.955283 17517 leader_election.cc:302] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b48418d221d64271985dd05d0ac39705, f5d1e341ccca4107afdfa11ae04719de; no voters:
I20260812 02:29:46.955628 17756 raft_consensus.cc:2454] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b48418d221d64271985dd05d0ac39705 in term 0.
I20260812 02:29:46.955765 17992 raft_consensus.cc:2781] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:46.955976 17992 raft_consensus.cc:482] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:46.956125 17992 raft_consensus.cc:3037] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:46.958860 17992 raft_consensus.cc:504] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.959585 17992 leader_election.cc:288] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [CANDIDATE]: Term 1 election: Requested vote from peers f5d1e341ccca4107afdfa11ae04719de (127.12.8.195:43097), 19f08323ab57425497b3a8f9ec1c55d1 (127.12.8.194:37755)
I20260812 02:29:46.960170 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34831398682543e3941566ca0f4c5532" candidate_uuid: "b48418d221d64271985dd05d0ac39705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f08323ab57425497b3a8f9ec1c55d1"
I20260812 02:29:46.960233 17908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34831398682543e3941566ca0f4c5532" candidate_uuid: "b48418d221d64271985dd05d0ac39705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d1e341ccca4107afdfa11ae04719de"
I20260812 02:29:46.960371 17756 raft_consensus.cc:3037] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:46.960487 17908 raft_consensus.cc:3037] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:46.962827 17756 raft_consensus.cc:2454] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48418d221d64271985dd05d0ac39705 in term 1.
I20260812 02:29:46.963145 17908 raft_consensus.cc:2454] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48418d221d64271985dd05d0ac39705 in term 1.
I20260812 02:29:46.963364 17518 leader_election.cc:302] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f08323ab57425497b3a8f9ec1c55d1, b48418d221d64271985dd05d0ac39705; no voters:
I20260812 02:29:46.963758 17992 raft_consensus.cc:2781] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:46.964356 17992 raft_consensus.cc:686] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 1 LEADER]: Becoming Leader. State: Replica: b48418d221d64271985dd05d0ac39705, State: Running, Role: LEADER
I20260812 02:29:46.964958 17992 consensus_queue.cc:237] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } }
I20260812 02:29:46.970836 17426 catalog_manager.cc:5495] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 reported cstate change: term changed from 0 to 1, leader changed from <none> to b48418d221d64271985dd05d0ac39705 (127.12.8.193). New cstate: current_term: 1 leader_uuid: "b48418d221d64271985dd05d0ac39705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48418d221d64271985dd05d0ac39705" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 40603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 } health_report { overall_health: UNKNOWN } } }
I20260812 02:29:46.993108 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:46.986943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.12.8.193:40603)
0812 02:29:46.987231 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:46.987237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:46.987259 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:29:46.987442 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:46.987446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:46.987557 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:46.987711 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:29:46.988168 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.988981 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.990496 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:46.990949 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:46.991097 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:46.991640 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:29:46.992435 (+ 795us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:29:46.992556 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:46.992678 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 02:29:46.992767 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 02:29:47.003420 17908 raft_consensus.cc:1261] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 1 FOLLOWER]: Refusing update from remote peer b48418d221d64271985dd05d0ac39705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:47.003466 17756 raft_consensus.cc:1261] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 1 FOLLOWER]: Refusing update from remote peer b48418d221d64271985dd05d0ac39705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:47.004704 17998 consensus_queue.cc:1035] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f08323ab57425497b3a8f9ec1c55d1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:47.005201 17992 consensus_queue.cc:1035] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d1e341ccca4107afdfa11ae04719de" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:29:47.013489 17654 tablet.cc:2369] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:47.019608 18001 mvcc.cc:205] Tried to move back new op lower bound from 7317511319552688128 to 7317511319413211136. Current Snapshot: MvccSnapshot[applied={T|T < 7317511319552688128}]
I20260812 02:29:47.038254 17426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:47.038535 17426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:47.042958 17426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e38838c15b494452b63095a4d5887dc7: Sending DeleteTablet for 3 replicas of tablet 34831398682543e3941566ca0f4c5532
I20260812 02:29:47.044211 17579 tablet_service.cc:1514] Processing DeleteTablet for tablet 34831398682543e3941566ca0f4c5532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:47 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:38440
I20260812 02:29:47.044471 17733 tablet_service.cc:1514] Processing DeleteTablet for tablet 34831398682543e3941566ca0f4c5532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:47 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:55988
I20260812 02:29:47.044550 17886 tablet_service.cc:1514] Processing DeleteTablet for tablet 34831398682543e3941566ca0f4c5532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:47 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:52082
I20260812 02:29:47.051311 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 17500
I20260812 02:29:47.059918 18013 tablet_replica.cc:330] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: stopping tablet replica
I20260812 02:29:47.060551 18013 raft_consensus.cc:2229] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:47.061221 18013 raft_consensus.cc:2258] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:47.064071 18013 ts_tablet_manager.cc:1905] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:47.079246 18014 tablet_replica.cc:330] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: stopping tablet replica
I20260812 02:29:47.080044 18014 raft_consensus.cc:2229] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:47.080896 18014 raft_consensus.cc:2258] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:47.084570 18013 ts_tablet_manager.cc:1918] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:29:47.084915 18013 log.cc:1196] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/34831398682543e3941566ca0f4c5532
I20260812 02:29:47.087872 18014 ts_tablet_manager.cc:1905] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:47.092824 18013 ts_tablet_manager.cc:1939] T 34831398682543e3941566ca0f4c5532 P 19f08323ab57425497b3a8f9ec1c55d1: Deleting consensus metadata
I20260812 02:29:47.093837 18011 tablet_replica.cc:330] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: stopping tablet replica
I20260812 02:29:47.094667 18011 raft_consensus.cc:2229] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:29:47.096818 18011 raft_consensus.cc:2258] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:47.097456 17410 catalog_manager.cc:4841] TS 19f08323ab57425497b3a8f9ec1c55d1 (127.12.8.194:37755): tablet 34831398682543e3941566ca0f4c5532 (table test-table [id=f93a2da9504f4d2ba4202484afe0700c]) successfully deleted
Aug 12 02:29:47 dist-test-slave-vq2q krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:29:47.104931 18014 ts_tablet_manager.cc:1918] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:29:47.105201 18014 log.cc:1196] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/34831398682543e3941566ca0f4c5532
I20260812 02:29:47.105765 18014 ts_tablet_manager.cc:1939] T 34831398682543e3941566ca0f4c5532 P f5d1e341ccca4107afdfa11ae04719de: Deleting consensus metadata
I20260812 02:29:47.107550 18011 ts_tablet_manager.cc:1905] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:47.108083 17409 catalog_manager.cc:4841] TS f5d1e341ccca4107afdfa11ae04719de (127.12.8.195:43097): tablet 34831398682543e3941566ca0f4c5532 (table test-table [id=f93a2da9504f4d2ba4202484afe0700c]) successfully deleted
I20260812 02:29:47.114209 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:47.053401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.12.8.193:40603)
0812 02:29:47.053639 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:47.053647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:47.075182 (+ 21535us) server_negotiation.cc:407] Connection header received
0812 02:29:47.075989 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:47.075997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:47.076121 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:47.076213 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:47.081319 (+ 5106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:47.082722 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:47.089534 (+ 6812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:47.090429 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:47.090587 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:47.103192 (+ 12605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:47.103224 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:47.103250 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:47.103326 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:47.108826 (+ 5500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:47.110443 (+ 1617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:47.110457 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:47.110461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:47.110739 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:47.111291 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:47.111298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:47.111301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:47.112045 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:47.112514 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:47.113358 (+ 844us) server_negotiation.cc:299] Negotiation successful
0812 02:29:47.113722 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:29:47.114447 18011 ts_tablet_manager.cc:1918] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:29:47.114646 18011 log.cc:1196] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/34831398682543e3941566ca0f4c5532
I20260812 02:29:47.115113 18011 ts_tablet_manager.cc:1939] T 34831398682543e3941566ca0f4c5532 P b48418d221d64271985dd05d0ac39705: Deleting consensus metadata
I20260812 02:29:47.116107 17647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:47.116916 17408 catalog_manager.cc:4841] TS b48418d221d64271985dd05d0ac39705 (127.12.8.193:40603): tablet 34831398682543e3941566ca0f4c5532 (table test-table [id=f93a2da9504f4d2ba4202484afe0700c]) successfully deleted
I20260812 02:29:47.233754 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 17500
I20260812 02:29:47.262028 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 17659
Aug 12 02:29:47 dist-test-slave-vq2q krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:29:47.277935 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:47.262873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.12.8.194:37755)
0812 02:29:47.263133 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:47.263140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:47.263161 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:29:47.263223 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:47.263226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:47.263325 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:47.263414 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:47.264699 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:47.265520 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:47.266790 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:47.267198 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:47.267315 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:47.271801 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:47.271817 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:47.271834 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:47.271872 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:47.275042 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:47.275744 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:47.275751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:47.275754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:47.275892 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:47.276261 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:47.276266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:47.276268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:47.276695 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:47.276993 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:47.277349 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 02:29:47.277581 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 02:29:47.278981 17799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:47.385968 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 17659
I20260812 02:29:47.410071 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 17812
Aug 12 02:29:47 dist-test-slave-vq2q krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:29:47.428047 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:47.411151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.12.8.195:43097)
0812 02:29:47.411376 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:47.411381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:47.411408 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:29:47.411557 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:47.411562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:47.411639 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:47.411775 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:47.413417 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:47.414178 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:47.415726 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:47.416142 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:47.416255 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:47.421473 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:47.421493 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:47.421511 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:47.421558 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:47.424744 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:47.425715 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:47.425772 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:47.425775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:47.425972 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:47.426387 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:47.426390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:47.426392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:47.426818 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:47.427052 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:47.427448 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 02:29:47.427648 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 02:29:47.428701 17959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:47.528338 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 17812
I20260812 02:29:47.550505 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 17389
I20260812 02:29:47.552248 17465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:47.668011 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 17389
2026-08-12T02:29:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4798 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T02:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:47Z Disabled control of system clock
I20260812 02:29:47.770054 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45525
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 02:29:48.205103 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:48.205626 18024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:48.205803 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:48.214339 18024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:48.214517 18024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:48.214599 18024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:48.214690 18024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:48.214778 18024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:48.227607 18024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:45525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:48.229504 18024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:48.231915 18024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:48.246150 18037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:48.246366 18033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:48.246670 18031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:48.247736 18024 server_base.cc:1034] running on GCE node
I20260812 02:29:48.248863 18024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:48.250895 18024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:48.252192 18024 hybrid_clock.cc:648] HybridClock initialized: now 1786501788252121 us; error 53 us; skew 500 ppm
I20260812 02:29:48.252691 18024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:29:48.255838 18024 webserver.cc:460] Webserver started at http://127.12.8.254:45067/ using document root <none> and password file <none>
I20260812 02:29:48.256827 18024 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:48.256934 18024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:48.257400 18024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:48.261134 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "6c111aa65fc24bb19363104aa9c7740a"
format_stamp: "Formatted at 2026-08-12 02:29:48 on dist-test-slave-vq2q"
server_key: "3f676b35cdd138292cbe2e0acb5615cb"
server_key_iv: "9341fda89f926012170b4518a8e1d403"
server_key_version: "encryptionkey@0"
I20260812 02:29:48.262149 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "6c111aa65fc24bb19363104aa9c7740a"
format_stamp: "Formatted at 2026-08-12 02:29:48 on dist-test-slave-vq2q"
server_key: "3f676b35cdd138292cbe2e0acb5615cb"
server_key_iv: "9341fda89f926012170b4518a8e1d403"
server_key_version: "encryptionkey@0"
I20260812 02:29:48.268939 18024 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 02:29:48.273494 18044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:48.275411 18024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:29:48.275637 18024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "6c111aa65fc24bb19363104aa9c7740a"
format_stamp: "Formatted at 2026-08-12 02:29:48 on dist-test-slave-vq2q"
server_key: "3f676b35cdd138292cbe2e0acb5615cb"
server_key_iv: "9341fda89f926012170b4518a8e1d403"
server_key_version: "encryptionkey@0"
I20260812 02:29:48.275899 18024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:48.283207 18024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:48.284224 18024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:48.284545 18024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:48.306612 18024 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:45525
I20260812 02:29:48.306646 18104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:45525 every 8 connection(s)
I20260812 02:29:48.308426 18024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:48.312892 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:48.318212 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 18024
I20260812 02:29:48.318477 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:48.319248 12323 external_mini_cluster.cc:1428] Setting key 154d411fe7fb120306940420e17c3fe1
I20260812 02:29:48.325773 18105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a: Bootstrap starting.
I20260812 02:29:48.330119 18105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:48.330701 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:48.322163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.12.8.254:45525)
0812 02:29:48.323606 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:48.323625 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:48.323689 (+ 64us) server_negotiation.cc:407] Connection header received
0812 02:29:48.326118 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:48.326152 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:48.326731 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:48.327212 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:48.328494 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:48.328503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:48.328520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:48.328540 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:48.328625 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:48.328649 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:48.328909 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:48.329048 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 02:29:48.329365 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":437,"threads_started":1}
I20260812 02:29:48.331511 18105 log.cc:824] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:48.335343 18105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a: No bootstrap required, opened a new log
I20260812 02:29:48.344048 18105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c111aa65fc24bb19363104aa9c7740a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45525 } }
I20260812 02:29:48.344653 18105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:48.344777 18105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c111aa65fc24bb19363104aa9c7740a, State: Initialized, Role: FOLLOWER
I20260812 02:29:48.345741 18105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c111aa65fc24bb19363104aa9c7740a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45525 } }
I20260812 02:29:48.346086 18105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:48.346267 18105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:48.346465 18105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:48.349033 18105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c111aa65fc24bb19363104aa9c7740a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45525 } }
I20260812 02:29:48.349730 18105 leader_election.cc:302] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c111aa65fc24bb19363104aa9c7740a; no voters:
I20260812 02:29:48.350459 18105 leader_election.cc:288] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:48.350720 18112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:48.351692 18112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [term 1 LEADER]: Becoming Leader. State: Replica: 6c111aa65fc24bb19363104aa9c7740a, State: Running, Role: LEADER
I20260812 02:29:48.352362 18112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c111aa65fc24bb19363104aa9c7740a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45525 } }
I20260812 02:29:48.352919 18105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:48.356047 18113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c111aa65fc24bb19363104aa9c7740a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c111aa65fc24bb19363104aa9c7740a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45525 } } }
I20260812 02:29:48.356061 18114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c111aa65fc24bb19363104aa9c7740a. Latest consensus state: current_term: 1 leader_uuid: "6c111aa65fc24bb19363104aa9c7740a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c111aa65fc24bb19363104aa9c7740a" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45525 } } }
I20260812 02:29:48.356649 18113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:48.356715 18114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:48.357944 18121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:48.364604 18121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:48.370805 18129 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:29:48.376682 18121 catalog_manager.cc:1347] Generated new cluster ID: fb968aa79c1046678a71d185656817cd
I20260812 02:29:48.376811 18121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:48.401448 18121 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:48.401961 18121 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:48.413015 18121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a: Generated new TSK 0
I20260812 02:29:48.414052 18121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:29:48.420718 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45525
--builtin_ntp_servers=127.12.8.212:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 02:29:48.865077 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:48.865540 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:48.865696 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:48.874001 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:48.874158 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:48.874323 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:48.885712 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:48.887575 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:48.890077 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:48.907095 18139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:48.907310 18140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:48.907369 18142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:48.908452 18131 server_base.cc:1034] running on GCE node
I20260812 02:29:48.909237 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:48.910393 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:48.911657 18131 hybrid_clock.cc:648] HybridClock initialized: now 1786501788911592 us; error 56 us; skew 500 ppm
I20260812 02:29:48.912055 18131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:29:48.915162 18131 webserver.cc:460] Webserver started at http://127.12.8.193:35087/ using document root <none> and password file <none>
I20260812 02:29:48.916208 18131 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:48.916327 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:48.916703 18131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:48.920583 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "3abf75a5af444b3fab4275a0f32c1e49"
format_stamp: "Formatted at 2026-08-12 02:29:48 on dist-test-slave-vq2q"
server_key: "b9968a979b4184edfa6b6a6fd4752943"
server_key_iv: "89821e07644fcabd23761efe95216a86"
server_key_version: "encryptionkey@0"
I20260812 02:29:48.921656 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "3abf75a5af444b3fab4275a0f32c1e49"
format_stamp: "Formatted at 2026-08-12 02:29:48 on dist-test-slave-vq2q"
server_key: "b9968a979b4184edfa6b6a6fd4752943"
server_key_iv: "89821e07644fcabd23761efe95216a86"
server_key_version: "encryptionkey@0"
I20260812 02:29:48.928862 18131 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 02:29:48.933756 18149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:48.935705 18131 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:29:48.935950 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "3abf75a5af444b3fab4275a0f32c1e49"
format_stamp: "Formatted at 2026-08-12 02:29:48 on dist-test-slave-vq2q"
server_key: "b9968a979b4184edfa6b6a6fd4752943"
server_key_iv: "89821e07644fcabd23761efe95216a86"
server_key_version: "encryptionkey@0"
I20260812 02:29:48.936267 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:48.943274 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:48.944329 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:48.944634 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:48.945895 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:48.947741 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:48.947854 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:48.947984 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:48.948072 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:48.987911 18131 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:39667
I20260812 02:29:48.987978 18277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:39667 every 8 connection(s)
I20260812 02:29:48.989632 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:48.997694 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 18131
I20260812 02:29:48.997922 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:48.998762 12323 external_mini_cluster.cc:1428] Setting key 93bca0bdb16baec7d0414045fe5f0369
I20260812 02:29:48.999744 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:48.992734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:50523 (local address 127.12.8.254:45525)
0812 02:29:48.993293 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:48.993298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:48.994782 (+ 1484us) server_negotiation.cc:407] Connection header received
0812 02:29:48.996710 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:48.996716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:48.996786 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:48.996888 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:48.998649 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:48.998656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:48.998660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:48.998686 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:48.998752 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:48.998767 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:48.998937 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:48.999218 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 02:29:48.999297 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
I20260812 02:29:49.000641 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:48.993403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45525 (local address 127.12.8.193:50523)
0812 02:29:48.994594 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:48.994647 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:48.996437 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:48.997192 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:48.997205 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:48.997335 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:48.998184 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:29:48.998187 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:48.998245 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:29:48.998367 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:48.998905 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:48.998917 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:48.998926 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:29:48.999248 (+ 322us) client_negotiation.cc:240] Negotiation successful
0812 02:29:48.999542 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":385,"threads_started":1}
I20260812 02:29:49.002521 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45525
--builtin_ntp_servers=127.12.8.212:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 02:29:49.002552 18278 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45525
I20260812 02:29:49.002965 18278 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:49.003782 18278 heartbeater.cc:511] Master 127.12.8.254:45525 requested a full tablet report, sending...
I20260812 02:29:49.006597 18064 ts_manager.cc:194] Registered new tserver with Master: 3abf75a5af444b3fab4275a0f32c1e49 (127.12.8.193:39667)
W20260812 02:29:49.369112 18282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:49.369472 18282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:49.369587 18282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:49.376919 18282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:49.377028 18282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:49.377162 18282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:49.386571 18282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:49.387944 18282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:49.389806 18282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:49.404062 18288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:49.404167 18290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:49.404148 18287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:49.405120 18282 server_base.cc:1034] running on GCE node
I20260812 02:29:49.405989 18282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:49.407200 18282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:49.408439 18282 hybrid_clock.cc:648] HybridClock initialized: now 1786501789408367 us; error 62 us; skew 500 ppm
I20260812 02:29:49.408828 18282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:29:49.411845 18282 webserver.cc:460] Webserver started at http://127.12.8.194:38993/ using document root <none> and password file <none>
I20260812 02:29:49.412669 18282 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:49.412774 18282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:49.413103 18282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:49.416201 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "9bafe5ae0eb34e3bb4da764bf4935d90"
format_stamp: "Formatted at 2026-08-12 02:29:49 on dist-test-slave-vq2q"
server_key: "900fd983e875e49513665b00cf58de64"
server_key_iv: "468e6d67b5bc2792d8580725049e9f33"
server_key_version: "encryptionkey@0"
I20260812 02:29:49.417016 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "9bafe5ae0eb34e3bb4da764bf4935d90"
format_stamp: "Formatted at 2026-08-12 02:29:49 on dist-test-slave-vq2q"
server_key: "900fd983e875e49513665b00cf58de64"
server_key_iv: "468e6d67b5bc2792d8580725049e9f33"
server_key_version: "encryptionkey@0"
I20260812 02:29:49.422819 18282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 02:29:49.426992 18297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:49.428617 18282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:49.428810 18282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "9bafe5ae0eb34e3bb4da764bf4935d90"
format_stamp: "Formatted at 2026-08-12 02:29:49 on dist-test-slave-vq2q"
server_key: "900fd983e875e49513665b00cf58de64"
server_key_iv: "468e6d67b5bc2792d8580725049e9f33"
server_key_version: "encryptionkey@0"
I20260812 02:29:49.429030 18282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:49.435138 18282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:49.435990 18282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:49.436239 18282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:49.437306 18282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:49.438803 18282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:49.438906 18282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:49.439011 18282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:49.439082 18282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:49.475677 18282 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:46815
I20260812 02:29:49.475780 18426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:46815 every 8 connection(s)
I20260812 02:29:49.477002 18282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:49.482260 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 18282
I20260812 02:29:49.482515 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:49.483081 12323 external_mini_cluster.cc:1428] Setting key ba25f3a9c25fcebf394c712ae572f44e
I20260812 02:29:49.486217 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45525
--builtin_ntp_servers=127.12.8.212:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 02:29:49.486368 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.480093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:55415 (local address 127.12.8.254:45525)
0812 02:29:49.480309 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:49.480315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:49.481782 (+ 1467us) server_negotiation.cc:407] Connection header received
0812 02:29:49.484118 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:49.484125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:49.484218 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:49.484307 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:49.485613 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:49.485619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:49.485624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:49.485643 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:49.485678 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:49.485690 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:49.485824 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:49.485998 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 02:29:49.486069 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:29:49.487327 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.480580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45525 (local address 127.12.8.194:55415)
0812 02:29:49.481637 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:49.481675 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:49.483858 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:49.484556 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:49.484564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:49.484654 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:49.485322 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:29:49.485326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:49.485365 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:29:49.485447 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:49.485782 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:49.485794 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:49.485803 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:29:49.486081 (+ 278us) client_negotiation.cc:240] Negotiation successful
0812 02:29:49.486348 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":303,"threads_started":1}
I20260812 02:29:49.488664 18427 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45525
I20260812 02:29:49.488942 18427 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:49.489562 18427 heartbeater.cc:511] Master 127.12.8.254:45525 requested a full tablet report, sending...
I20260812 02:29:49.490952 18064 ts_manager.cc:194] Registered new tserver with Master: 9bafe5ae0eb34e3bb4da764bf4935d90 (127.12.8.194:46815)
W20260812 02:29:49.829502 18430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:49.829836 18430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:49.829936 18430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:49.837512 18430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:49.837623 18430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:49.837734 18430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:29:49.847199 18430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:49.848536 18430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:49.850411 18430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:49.863514 18436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:49.863566 18435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:49.863590 18438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:49.864328 18430 server_base.cc:1034] running on GCE node
I20260812 02:29:49.864912 18430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:49.865934 18430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:49.867144 18430 hybrid_clock.cc:648] HybridClock initialized: now 1786501789867087 us; error 47 us; skew 500 ppm
I20260812 02:29:49.867470 18430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:29:49.869732 18430 webserver.cc:460] Webserver started at http://127.12.8.195:36129/ using document root <none> and password file <none>
I20260812 02:29:49.870414 18430 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:49.870512 18430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:49.870836 18430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:49.873734 18430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "20761fe5ac22475e8329adbab987018e"
format_stamp: "Formatted at 2026-08-12 02:29:49 on dist-test-slave-vq2q"
server_key: "ce93cfd3f6446b6fa7d11b7f6bf5e091"
server_key_iv: "6772938db628dcef31a96307095ff385"
server_key_version: "encryptionkey@0"
I20260812 02:29:49.874527 18430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "20761fe5ac22475e8329adbab987018e"
format_stamp: "Formatted at 2026-08-12 02:29:49 on dist-test-slave-vq2q"
server_key: "ce93cfd3f6446b6fa7d11b7f6bf5e091"
server_key_iv: "6772938db628dcef31a96307095ff385"
server_key_version: "encryptionkey@0"
I20260812 02:29:49.879921 18430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:29:49.883719 18445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:49.885097 18430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 02:29:49.885295 18430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "20761fe5ac22475e8329adbab987018e"
format_stamp: "Formatted at 2026-08-12 02:29:49 on dist-test-slave-vq2q"
server_key: "ce93cfd3f6446b6fa7d11b7f6bf5e091"
server_key_iv: "6772938db628dcef31a96307095ff385"
server_key_version: "encryptionkey@0"
I20260812 02:29:49.885522 18430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:49.891503 18430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:49.892383 18430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:49.892655 18430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:49.893666 18430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:49.895247 18430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:49.895357 18430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:49.895467 18430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:49.895537 18430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:49.927357 18430 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:44805
I20260812 02:29:49.927424 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:44805 every 8 connection(s)
I20260812 02:29:49.928752 18430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:29:49.932642 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 18430
I20260812 02:29:49.932868 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:29:49.933528 12323 external_mini_cluster.cc:1428] Setting key e4b9e5f9dc6e41458dfb315541dfcabb
I20260812 02:29:49.938223 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.931640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:58883 (local address 127.12.8.254:45525)
0812 02:29:49.931944 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:49.931949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:49.933603 (+ 1654us) server_negotiation.cc:407] Connection header received
0812 02:29:49.935940 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:49.935945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:49.936015 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:49.936129 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:49.937537 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:49.937542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:49.937545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:49.937564 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:49.937598 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:49.937611 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:49.937712 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:49.937942 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 02:29:49.937993 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 02:29:49.939253 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.932151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45525 (local address 127.12.8.195:58883)
0812 02:29:49.933419 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:49.933467 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:49.935598 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:49.936314 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:49.936323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:49.936418 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:49.937157 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:29:49.937160 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:49.937250 (+ 90us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:29:49.937352 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:49.937728 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:49.937739 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:49.937746 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:29:49.938009 (+ 263us) client_negotiation.cc:240] Negotiation successful
0812 02:29:49.938224 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":354,"threads_started":1}
I20260812 02:29:49.940793 18577 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45525
I20260812 02:29:49.941123 18577 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:49.941849 18577 heartbeater.cc:511] Master 127.12.8.254:45525 requested a full tablet report, sending...
I20260812 02:29:49.943568 18064 ts_manager.cc:194] Registered new tserver with Master: 20761fe5ac22475e8329adbab987018e (127.12.8.195:44805)
I20260812 02:29:49.949815 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:29:49.957523 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.954843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.12.8.254:45525)
0812 02:29:49.955026 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:49.955031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:49.955585 (+ 554us) server_negotiation.cc:407] Connection header received
0812 02:29:49.955673 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:49.955677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:49.955747 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:49.955833 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:49.956803 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:49.956807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:49.956811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:49.956826 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:49.956861 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:49.956873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:49.956978 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:49.957169 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 02:29:49.957280 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:29:49.961807 18064 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:29:49.968758 18064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:29:49.987051 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.981360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:46815 (local address 127.0.0.1:54958)
0812 02:29:49.983048 (+ 1688us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:49.983065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:49.983164 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:49.984769 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:49.984774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:49.984798 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:49.985930 (+ 1132us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:29:49.985933 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:49.985950 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:29:49.985993 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:49.986379 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:49.986383 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:49.986387 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 02:29:49.986520 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 02:29:49.986568 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1550,"thread_start_us":1411,"threads_started":1}
I20260812 02:29:49.987015 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.980977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45198)
0812 02:29:49.981901 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:49.981955 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:49.982100 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:49.983779 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:49.983787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:49.983885 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:49.984818 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:29:49.984821 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:49.984852 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:29:49.984963 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:49.986149 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:49.986161 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:49.986169 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:29:49.986335 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 02:29:49.986432 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":355,"threads_started":1}
I20260812 02:29:49.988090 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.981235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.12.8.193:39667)
0812 02:29:49.983151 (+ 1916us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:49.983157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:49.983183 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:29:49.983287 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:49.983295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:49.983545 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:49.984462 (+ 917us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:49.985901 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:49.985909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:49.985922 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:49.985940 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:49.985999 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:49.986024 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:49.987212 (+ 1188us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:49.987551 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 02:29:49.987652 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1754,"thread_start_us":1706,"threads_started":1}
I20260812 02:29:49.988505 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.981979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.12.8.194:46815)
0812 02:29:49.982617 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:49.982626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:49.983290 (+ 664us) server_negotiation.cc:407] Connection header received
0812 02:29:49.983372 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:49.983380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:49.983590 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:49.983761 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:49.986155 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:49.986163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:49.986175 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:49.986196 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:49.986256 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:49.986276 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:49.986447 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:49.986512 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 02:29:49.986622 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":329,"threads_started":1}
I20260812 02:29:49.992087 18351 tablet_service.cc:1467] Processing CreateTablet for tablet 110b33b0e1554690bbebfb47baf214d1 (DEFAULT_TABLE table=test-table [id=4ccab44cb8734bc8bd09dd38e0c160ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:49.992131 18200 tablet_service.cc:1467] Processing CreateTablet for tablet 110b33b0e1554690bbebfb47baf214d1 (DEFAULT_TABLE table=test-table [id=4ccab44cb8734bc8bd09dd38e0c160ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:49.993368 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.983766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.12.8.195:44805)
0812 02:29:49.985134 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:49.985141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:49.985854 (+ 713us) server_negotiation.cc:407] Connection header received
0812 02:29:49.985929 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:49.985936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:49.986143 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:49.986278 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:49.992498 (+ 6220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:49.992506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:49.992515 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:49.992532 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:49.992576 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:49.992590 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:49.992792 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:49.992859 (+ 67us) server_negotiation.cc:299] Negotiation successful
0812 02:29:49.992955 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":993,"threads_started":1}
I20260812 02:29:49.993590 18351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b33b0e1554690bbebfb47baf214d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:49.993641 18200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b33b0e1554690bbebfb47baf214d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:49.994215 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:49.983176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:44805 (local address 127.0.0.1:55980)
0812 02:29:49.985537 (+ 2361us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:49.985554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:49.985659 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:49.987728 (+ 2069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:49.987732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:49.987755 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:49.992258 (+ 4503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:29:49.992261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:49.992278 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:29:49.992320 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:49.992699 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:49.992704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:49.992709 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:29:49.992831 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 02:29:49.992890 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2228,"thread_start_us":2145,"threads_started":1}
I20260812 02:29:49.997592 18502 tablet_service.cc:1467] Processing CreateTablet for tablet 110b33b0e1554690bbebfb47baf214d1 (DEFAULT_TABLE table=test-table [id=4ccab44cb8734bc8bd09dd38e0c160ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:29:49.999087 18502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b33b0e1554690bbebfb47baf214d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:50.008963 18596 tablet_bootstrap.cc:492] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: Bootstrap starting.
I20260812 02:29:50.009928 18595 tablet_bootstrap.cc:492] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Bootstrap starting.
I20260812 02:29:50.010813 18278 heartbeater.cc:503] Master 127.12.8.254:45525 was elected leader, sending a full tablet report...
I20260812 02:29:50.011142 18597 tablet_bootstrap.cc:492] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Bootstrap starting.
I20260812 02:29:50.015012 18595 tablet_bootstrap.cc:654] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:50.015128 18597 tablet_bootstrap.cc:654] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:50.015131 18596 tablet_bootstrap.cc:654] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:50.016216 18595 log.cc:824] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:50.016384 18597 log.cc:824] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:50.016372 18596 log.cc:824] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:50.019214 18595 tablet_bootstrap.cc:492] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: No bootstrap required, opened a new log
I20260812 02:29:50.019454 18596 tablet_bootstrap.cc:492] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: No bootstrap required, opened a new log
I20260812 02:29:50.019456 18597 tablet_bootstrap.cc:492] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: No bootstrap required, opened a new log
I20260812 02:29:50.019500 18595 ts_tablet_manager.cc:1397] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260812 02:29:50.019719 18597 ts_tablet_manager.cc:1397] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 02:29:50.019719 18596 ts_tablet_manager.cc:1397] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260812 02:29:50.027232 18595 raft_consensus.cc:348] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.027649 18595 raft_consensus.cc:374] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:50.027762 18595 raft_consensus.cc:729] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bafe5ae0eb34e3bb4da764bf4935d90, State: Initialized, Role: FOLLOWER
I20260812 02:29:50.028395 18595 consensus_queue.cc:260] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.029515 18427 heartbeater.cc:503] Master 127.12.8.254:45525 was elected leader, sending a full tablet report...
I20260812 02:29:50.030172 18596 raft_consensus.cc:348] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.030172 18597 raft_consensus.cc:348] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.030630 18597 raft_consensus.cc:374] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:50.030704 18596 raft_consensus.cc:374] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:50.030787 18597 raft_consensus.cc:729] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20761fe5ac22475e8329adbab987018e, State: Initialized, Role: FOLLOWER
I20260812 02:29:50.030857 18596 raft_consensus.cc:729] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3abf75a5af444b3fab4275a0f32c1e49, State: Initialized, Role: FOLLOWER
I20260812 02:29:50.031489 18595 ts_tablet_manager.cc:1428] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260812 02:29:50.031571 18597 consensus_queue.cc:260] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.031571 18596 consensus_queue.cc:260] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.033419 18577 heartbeater.cc:503] Master 127.12.8.254:45525 was elected leader, sending a full tablet report...
I20260812 02:29:50.033746 18596 ts_tablet_manager.cc:1428] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260812 02:29:50.034394 18597 ts_tablet_manager.cc:1428] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
W20260812 02:29:50.180586 18578 tablet.cc:2369] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:50.219261 18602 raft_consensus.cc:482] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:29:50.219563 18602 raft_consensus.cc:504] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.221611 18602 leader_election.cc:288] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3abf75a5af444b3fab4275a0f32c1e49 (127.12.8.193:39667), 20761fe5ac22475e8329adbab987018e (127.12.8.195:44805)
I20260812 02:29:50.225059 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.221971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:55811 (local address 127.12.8.193:39667)
0812 02:29:50.222183 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:50.222190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:50.222791 (+ 601us) server_negotiation.cc:407] Connection header received
0812 02:29:50.222961 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:50.222968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:50.223043 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:50.223152 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:50.224421 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:50.224428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:50.224432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:50.224452 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:50.224493 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:50.224509 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:50.224633 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:50.224749 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 02:29:50.224808 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 02:29:50.225271 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.221973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.12.8.194:55811)
0812 02:29:50.222684 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.222700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:50.222825 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:50.223302 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:50.223308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:50.223338 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:50.224153 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:29:50.224156 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:50.224178 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:29:50.224231 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:50.224622 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:50.224628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:50.224634 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:29:50.224819 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 02:29:50.224874 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":409,"threads_started":1}
I20260812 02:29:50.225845 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.222994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:51997 (local address 127.12.8.195:44805)
0812 02:29:50.223173 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:50.223179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:50.223646 (+ 467us) server_negotiation.cc:407] Connection header received
0812 02:29:50.223785 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:50.223790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:50.223851 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:50.223952 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:50.224996 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:50.225002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:50.225006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:50.225025 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:50.225067 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:50.225084 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:50.225244 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:50.225467 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 02:29:50.225532 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:29:50.225946 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.222811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:44805 (local address 127.12.8.194:51997)
0812 02:29:50.223538 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.223559 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:50.223673 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:50.223993 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:50.224003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:50.224021 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:50.224727 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:29:50.224730 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:50.224801 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:29:50.224846 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:50.225326 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:50.225331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:50.225337 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:29:50.225482 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 02:29:50.225531 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":385,"threads_started":1}
I20260812 02:29:50.226306 18224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b33b0e1554690bbebfb47baf214d1" candidate_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3abf75a5af444b3fab4275a0f32c1e49" is_pre_election: true
I20260812 02:29:50.226962 18224 raft_consensus.cc:2454] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bafe5ae0eb34e3bb4da764bf4935d90 in term 0.
I20260812 02:29:50.226907 18522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b33b0e1554690bbebfb47baf214d1" candidate_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20761fe5ac22475e8329adbab987018e" is_pre_election: true
I20260812 02:29:50.227407 18522 raft_consensus.cc:2454] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bafe5ae0eb34e3bb4da764bf4935d90 in term 0.
I20260812 02:29:50.227701 18300 leader_election.cc:302] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3abf75a5af444b3fab4275a0f32c1e49, 9bafe5ae0eb34e3bb4da764bf4935d90; no voters:
I20260812 02:29:50.228109 18602 raft_consensus.cc:2781] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:29:50.228264 18602 raft_consensus.cc:482] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:29:50.228379 18602 raft_consensus.cc:3037] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 0 FOLLOWER]: Advancing to term 1
W20260812 02:29:50.229128 18428 tablet.cc:2369] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:50.230791 18602 raft_consensus.cc:504] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.231416 18602 leader_election.cc:288] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [CANDIDATE]: Term 1 election: Requested vote from peers 3abf75a5af444b3fab4275a0f32c1e49 (127.12.8.193:39667), 20761fe5ac22475e8329adbab987018e (127.12.8.195:44805)
I20260812 02:29:50.231771 18224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b33b0e1554690bbebfb47baf214d1" candidate_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3abf75a5af444b3fab4275a0f32c1e49"
I20260812 02:29:50.231866 18522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b33b0e1554690bbebfb47baf214d1" candidate_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20761fe5ac22475e8329adbab987018e"
I20260812 02:29:50.231953 18224 raft_consensus.cc:3037] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:50.232064 18522 raft_consensus.cc:3037] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:50.234381 18522 raft_consensus.cc:2454] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bafe5ae0eb34e3bb4da764bf4935d90 in term 1.
I20260812 02:29:50.234408 18224 raft_consensus.cc:2454] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bafe5ae0eb34e3bb4da764bf4935d90 in term 1.
I20260812 02:29:50.234750 18300 leader_election.cc:302] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20761fe5ac22475e8329adbab987018e, 9bafe5ae0eb34e3bb4da764bf4935d90; no voters:
I20260812 02:29:50.235040 18602 raft_consensus.cc:2781] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:50.235510 18602 raft_consensus.cc:686] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 1 LEADER]: Becoming Leader. State: Replica: 9bafe5ae0eb34e3bb4da764bf4935d90, State: Running, Role: LEADER
I20260812 02:29:50.236078 18602 consensus_queue.cc:237] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } }
I20260812 02:29:50.240725 18063 catalog_manager.cc:5495] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bafe5ae0eb34e3bb4da764bf4935d90 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bafe5ae0eb34e3bb4da764bf4935d90" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 } health_report { overall_health: UNKNOWN } } }
W20260812 02:29:50.242436 18279 tablet.cc:2369] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:29:50.287904 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.285920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.12.8.194:46815)
0812 02:29:50.286101 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:50.286107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:50.286126 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:29:50.286304 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:50.286309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:50.286391 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:50.286482 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:50.287322 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:50.287326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:50.287329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:50.287344 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:50.287376 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:50.287388 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:50.287506 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:50.287587 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 02:29:50.287648 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:29:50.296237 18522 raft_consensus.cc:1261] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 1 FOLLOWER]: Refusing update from remote peer 9bafe5ae0eb34e3bb4da764bf4935d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:50.296237 18224 raft_consensus.cc:1261] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 1 FOLLOWER]: Refusing update from remote peer 9bafe5ae0eb34e3bb4da764bf4935d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:29:50.297196 18608 consensus_queue.cc:1035] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3abf75a5af444b3fab4275a0f32c1e49" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:50.297590 18602 consensus_queue.cc:1035] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20761fe5ac22475e8329adbab987018e" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:29:50.308043 18611 mvcc.cc:205] Tried to move back new op lower bound from 7317511333043826688 to 7317511332810633216. Current Snapshot: MvccSnapshot[applied={T|T < 7317511333043826688}]
I20260812 02:29:50.313354 18615 mvcc.cc:205] Tried to move back new op lower bound from 7317511333043826688 to 7317511332810633216. Current Snapshot: MvccSnapshot[applied={T|T < 7317511333043826688}]
I20260812 02:29:50.324195 18063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:50.324631 18063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:29:50.329000 18063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c111aa65fc24bb19363104aa9c7740a: Sending DeleteTablet for 3 replicas of tablet 110b33b0e1554690bbebfb47baf214d1
I20260812 02:29:50.330439 18351 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b33b0e1554690bbebfb47baf214d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:50 UTC) from {username='slave'} at 127.0.0.1:54958
I20260812 02:29:50.330447 18200 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b33b0e1554690bbebfb47baf214d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:50 UTC) from {username='slave'} at 127.0.0.1:45198
I20260812 02:29:50.330479 18502 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b33b0e1554690bbebfb47baf214d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:29:50 UTC) from {username='slave'} at 127.0.0.1:55980
I20260812 02:29:50.333243 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 18131
I20260812 02:29:50.342438 18625 tablet_replica.cc:330] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: stopping tablet replica
I20260812 02:29:50.343042 18625 raft_consensus.cc:2229] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:29:50.343818 18625 raft_consensus.cc:2258] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:50.344813 18624 tablet_replica.cc:330] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: stopping tablet replica
I20260812 02:29:50.345347 18624 raft_consensus.cc:2229] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:50.345886 18626 tablet_replica.cc:330] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: stopping tablet replica
I20260812 02:29:50.345996 18624 raft_consensus.cc:2258] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:50.346341 18626 raft_consensus.cc:2229] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:29:50.346870 18626 raft_consensus.cc:2258] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:29:50.348410 18624 ts_tablet_manager.cc:1905] T 110b33b0e1554690bbebfb47baf214d1 P 3abf75a5af444b3fab4275a0f32c1e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:50.348474 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.334931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.12.8.193:39667)
0812 02:29:50.335103 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:50.335109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:50.345082 (+ 9973us) server_negotiation.cc:407] Connection header received
0812 02:29:50.345336 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:50.345342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:50.345464 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:50.345574 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:50.347417 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:50.347423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:50.347428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:50.347452 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:50.347496 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:50.347515 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:50.347636 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:50.348106 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 02:29:50.348172 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:29:50.348945 18626 ts_tablet_manager.cc:1905] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:50.349390 18271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:50.349876 18625 ts_tablet_manager.cc:1905] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:29:50.356259 18626 ts_tablet_manager.cc:1918] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:29:50.356524 18626 log.cc:1196] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/110b33b0e1554690bbebfb47baf214d1
I20260812 02:29:50.357141 18626 ts_tablet_manager.cc:1939] T 110b33b0e1554690bbebfb47baf214d1 P 20761fe5ac22475e8329adbab987018e: Deleting consensus metadata
I20260812 02:29:50.359411 18047 catalog_manager.cc:4841] TS 20761fe5ac22475e8329adbab987018e (127.12.8.195:44805): tablet 110b33b0e1554690bbebfb47baf214d1 (table test-table [id=4ccab44cb8734bc8bd09dd38e0c160ad]) successfully deleted
I20260812 02:29:50.359745 18625 ts_tablet_manager.cc:1918] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:29:50.359987 18625 log.cc:1196] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/110b33b0e1554690bbebfb47baf214d1
I20260812 02:29:50.360561 18625 ts_tablet_manager.cc:1939] T 110b33b0e1554690bbebfb47baf214d1 P 9bafe5ae0eb34e3bb4da764bf4935d90: Deleting consensus metadata
I20260812 02:29:50.362625 18046 catalog_manager.cc:4841] TS 9bafe5ae0eb34e3bb4da764bf4935d90 (127.12.8.194:46815): tablet 110b33b0e1554690bbebfb47baf214d1 (table test-table [id=4ccab44cb8734bc8bd09dd38e0c160ad]) successfully deleted
I20260812 02:29:50.446888 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 18131
I20260812 02:29:50.468631 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 18282
W20260812 02:29:50.468608 18047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.8.193:39667 (error 108)
I20260812 02:29:50.470638 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.470083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45220)
0812 02:29:50.470325 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.470462 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260812 02:29:50.470919 18047 catalog_manager.cc:4584] TS 3abf75a5af444b3fab4275a0f32c1e49 (127.12.8.193:39667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 110b33b0e1554690bbebfb47baf214d1: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
I20260812 02:29:50.472110 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.469590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.12.8.194:46815)
0812 02:29:50.469813 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:50.469818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:50.470129 (+ 311us) server_negotiation.cc:407] Connection header received
0812 02:29:50.470366 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:50.470370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:50.470457 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:50.470557 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:50.471472 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:50.471478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:50.471483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:50.471499 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:50.471531 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:50.471542 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:50.471664 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:50.471768 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:29:50.471840 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:29:50.472872 18421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:50.525071 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.524336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45236)
0812 02:29:50.524667 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.524856 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260812 02:29:50.526582 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.526149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45250)
0812 02:29:50.526324 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.526431 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 02:29:50.588042 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.587274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45256)
0812 02:29:50.587620 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.587794 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260812 02:29:50.589591 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.589103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45272)
0812 02:29:50.589326 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.589435 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 02:29:50.597574 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 18282
I20260812 02:29:50.623462 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 18430
I20260812 02:29:50.649166 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.624551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.12.8.195:44805)
0812 02:29:50.624734 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:50.624739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:50.624762 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:29:50.624941 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:50.624945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:50.625057 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:50.625150 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:50.648199 (+ 23049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:50.648207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:50.648211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:50.648241 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:50.648293 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:50.648308 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:50.648475 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:50.648690 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 02:29:50.648771 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:29:50.650230 18572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:50.670547 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.669726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45284)
0812 02:29:50.670097 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.670272 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 02:29:50.671957 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.671430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45298)
0812 02:29:50.671679 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.671780 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260812 02:29:50.752411 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 18430
I20260812 02:29:50.772595 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 18024
I20260812 02:29:50.774121 18100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:50.901937 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:50.901108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39667 (local address 127.0.0.1:45310)
0812 02:29:50.901452 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:50.901601 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 02:29:50.902226 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 18024
2026-08-12T02:29:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3228 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 02:29:50.952689 12323 mini_oidc.cc:152] Starting JWKS server
I20260812 02:29:50.956564 12323 webserver.cc:460] Webserver started at https://127.0.0.1:36475/ using document root <none> and password file <none>
I20260812 02:29:50.957015 12323 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:29:50.957973 12323 webserver.cc:460] Webserver started at http://127.0.0.1:41785/ using document root <none> and password file <none>
2026-08-12T02:29:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:50Z Disabled control of system clock
I20260812 02:29:51.009647 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40915
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40915
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36475/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/testchainca.cert with env {}
W20260812 02:29:51.363011 18640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:51.363333 18640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:51.363461 18640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:51.370880 18640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:51.371018 18640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:51.371084 18640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:51.371176 18640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:51.371248 18640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:29:51.371341 18640 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:29:51.371407 18640 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36475/jwks.json
I20260812 02:29:51.382102 18640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36475/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:51.383481 18640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:51.384872 18640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:51.398277 18648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:51.398343 18651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:51.398379 18647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:51.399598 18640 server_base.cc:1034] running on GCE node
I20260812 02:29:51.400754 18640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:51.403021 18640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:51.404323 18640 hybrid_clock.cc:648] HybridClock initialized: now 1786501791404239 us; error 61 us; skew 500 ppm
I20260812 02:29:51.404820 18640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:29:51.408274 18640 webserver.cc:460] Webserver started at http://127.12.8.254:44105/ using document root <none> and password file <none>
I20260812 02:29:51.409320 18640 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:51.409459 18640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:51.409830 18640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:51.413950 18640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "bda1b636f8c94b23b06a8905e35ccbb5"
format_stamp: "Formatted at 2026-08-12 02:29:51 on dist-test-slave-vq2q"
server_key: "a043d29a5a9bdb74bb5426dba0e09a02"
server_key_iv: "1793a9fb46e3cee5c60a6df8d118a715"
server_key_version: "encryptionkey@0"
I20260812 02:29:51.415436 18640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "bda1b636f8c94b23b06a8905e35ccbb5"
format_stamp: "Formatted at 2026-08-12 02:29:51 on dist-test-slave-vq2q"
server_key: "a043d29a5a9bdb74bb5426dba0e09a02"
server_key_iv: "1793a9fb46e3cee5c60a6df8d118a715"
server_key_version: "encryptionkey@0"
I20260812 02:29:51.423992 18640 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260812 02:29:51.429011 18657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:51.430945 18640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:29:51.431178 18640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "bda1b636f8c94b23b06a8905e35ccbb5"
format_stamp: "Formatted at 2026-08-12 02:29:51 on dist-test-slave-vq2q"
server_key: "a043d29a5a9bdb74bb5426dba0e09a02"
server_key_iv: "1793a9fb46e3cee5c60a6df8d118a715"
server_key_version: "encryptionkey@0"
I20260812 02:29:51.431432 18640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:51.449790 18640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:51.450820 18640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:51.451139 18640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:51.472429 18640 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:40915
I20260812 02:29:51.472524 18712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:40915 every 8 connection(s)
I20260812 02:29:51.474196 18640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:51.478524 18714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:51.483184 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 18640
I20260812 02:29:51.483426 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:51.484127 12323 external_mini_cluster.cc:1428] Setting key 8a69f8b070b1f15e917e0cf18acab028
I20260812 02:29:51.490327 18714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5: Bootstrap starting.
I20260812 02:29:51.493968 18714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:51.495285 18714 log.cc:824] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:51.499521 18714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5: No bootstrap required, opened a new log
I20260812 02:29:51.501853 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:51.486577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.12.8.254:40915)
0812 02:29:51.487940 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:51.487954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:51.488016 (+ 62us) server_negotiation.cc:407] Connection header received
0812 02:29:51.490203 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:51.490235 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:51.490881 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:51.491307 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:51.492078 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:51.493606 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:51.495565 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:51.496444 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:51.496634 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:51.498091 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:51.498103 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:51.498127 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:51.499068 (+ 941us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:51.499216 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:51.499249 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:51.499418 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:51.499676 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 02:29:51.500137 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":476,"threads_started":1}
I20260812 02:29:51.507313 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:29:51.510064 18714 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1b636f8c94b23b06a8905e35ccbb5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40915 } }
I20260812 02:29:51.510675 18714 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:51.510799 18714 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda1b636f8c94b23b06a8905e35ccbb5, State: Initialized, Role: FOLLOWER
I20260812 02:29:51.511591 18714 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1b636f8c94b23b06a8905e35ccbb5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40915 } }
I20260812 02:29:51.511871 18714 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:51.511989 18714 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:51.512151 18714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:51.515187 18714 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1b636f8c94b23b06a8905e35ccbb5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40915 } }
I20260812 02:29:51.515854 18714 leader_election.cc:302] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda1b636f8c94b23b06a8905e35ccbb5; no voters:
I20260812 02:29:51.516554 18714 leader_election.cc:288] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:51.516815 18722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:51.517779 18722 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [term 1 LEADER]: Becoming Leader. State: Replica: bda1b636f8c94b23b06a8905e35ccbb5, State: Running, Role: LEADER
I20260812 02:29:51.518368 18722 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1b636f8c94b23b06a8905e35ccbb5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40915 } }
I20260812 02:29:51.519335 18714 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:51.522236 18723 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda1b636f8c94b23b06a8905e35ccbb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1b636f8c94b23b06a8905e35ccbb5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40915 } } }
I20260812 02:29:51.522800 18723 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:51.523731 18724 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda1b636f8c94b23b06a8905e35ccbb5. Latest consensus state: current_term: 1 leader_uuid: "bda1b636f8c94b23b06a8905e35ccbb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda1b636f8c94b23b06a8905e35ccbb5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40915 } } }
I20260812 02:29:51.525856 18724 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:51.526683 18731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:51.532420 18731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:51.542117 18731 catalog_manager.cc:1347] Generated new cluster ID: 5781f7d7e7b944a3a7e50c6accfa4599
I20260812 02:29:51.542225 18731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:51.557529 18731 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:51.558866 18731 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:51.570041 18731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda1b636f8c94b23b06a8905e35ccbb5: Generated new TSK 0
I20260812 02:29:51.570852 18731 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 02:29:51.611244 18748 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 02:29:51.611462 18748 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 02:29:51.612788 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:51.589594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.12.8.254:40915)
0812 02:29:51.589795 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:51.589802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:51.590397 (+ 595us) server_negotiation.cc:407] Connection header received
0812 02:29:51.590742 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:51.590749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:51.590840 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:51.591001 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:29:51.591745 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:51.592954 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:51.594440 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:51.595114 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:51.595235 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:51.595783 (+ 548us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:29:51.612057 (+ 16274us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 02:29:51.612477 (+ 420us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32806: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 02:29:51.612926 18718 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32806: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 02:29:51.614506 18747 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:29:51.589567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40915 (local address 127.0.0.1:32806)
0812 02:29:51.590295 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:51.590313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:51.590458 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:51.591056 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:51.591061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:51.591073 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:29:51.591559 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:51.591573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:51.593097 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:51.593101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:51.594298 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:51.594311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:51.595321 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:51.595326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:51.595459 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:51.595523 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:29:51.612401 (+ 16878us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 02:29:51.612701 (+ 300us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.8.254:40915: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":405,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.8.254:40915: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 02:29:51.616212 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 18640
I20260812 02:29:51.617412 18708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:51.712610 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 18640
2026-08-12T02:29:51Z chronyd exiting
I20260812 02:29:51.766062 12323 test_util.cc:175] -----------------------------------------------
I20260812 02:29:51.766199 12323 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501741404412-12323-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (819 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 02:29:51.770156 12323 mini_oidc.cc:152] Starting JWKS server
I20260812 02:29:51.772092 12323 webserver.cc:460] Webserver started at https://127.0.0.1:39057/ using document root <none> and password file <none>
I20260812 02:29:51.772332 12323 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:29:51.772965 12323 webserver.cc:460] Webserver started at http://127.0.0.1:36685/ using document root <none> and password file <none>
2026-08-12T02:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:51Z Disabled control of system clock
I20260812 02:29:51.809229 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46711
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:37679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:46711
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39057/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/testchainca.cert with env {}
W20260812 02:29:52.154608 18755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:52.154918 18755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:52.155038 18755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:52.162439 18755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:52.162573 18755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:52.162636 18755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:52.162709 18755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:52.162760 18755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:29:52.162848 18755 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:29:52.162905 18755 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39057/jwks.json
I20260812 02:29:52.173316 18755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:37679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:46711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39057/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:52.174721 18755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:52.176054 18755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:52.187651 18762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:52.187609 18763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:52.187566 18765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:52.188370 18755 server_base.cc:1034] running on GCE node
I20260812 02:29:52.189203 18755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:52.190663 18755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:52.191865 18755 hybrid_clock.cc:648] HybridClock initialized: now 1786501792191822 us; error 39 us; skew 500 ppm
I20260812 02:29:52.192191 18755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:29:52.194519 18755 webserver.cc:460] Webserver started at http://127.12.8.254:33567/ using document root <none> and password file <none>
I20260812 02:29:52.195214 18755 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:52.195307 18755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:52.195592 18755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:52.198540 18755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "b2e6f48b8b294e1896d7df9b22e01822"
format_stamp: "Formatted at 2026-08-12 02:29:52 on dist-test-slave-vq2q"
server_key: "290ebe6caad6e78862101d1fb1cf84a2"
server_key_iv: "1f7aa74e1bbb7b923667913ddb9e0d89"
server_key_version: "encryptionkey@0"
I20260812 02:29:52.199321 18755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "b2e6f48b8b294e1896d7df9b22e01822"
format_stamp: "Formatted at 2026-08-12 02:29:52 on dist-test-slave-vq2q"
server_key: "290ebe6caad6e78862101d1fb1cf84a2"
server_key_iv: "1f7aa74e1bbb7b923667913ddb9e0d89"
server_key_version: "encryptionkey@0"
I20260812 02:29:52.204773 18755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 02:29:52.208493 18774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:52.209898 18755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:29:52.210067 18755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "b2e6f48b8b294e1896d7df9b22e01822"
format_stamp: "Formatted at 2026-08-12 02:29:52 on dist-test-slave-vq2q"
server_key: "290ebe6caad6e78862101d1fb1cf84a2"
server_key_iv: "1f7aa74e1bbb7b923667913ddb9e0d89"
server_key_version: "encryptionkey@0"
I20260812 02:29:52.210291 18755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:52.228566 18755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:52.229475 18755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:52.229777 18755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:52.247571 18755 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:46711
I20260812 02:29:52.247593 18834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:46711 every 8 connection(s)
I20260812 02:29:52.248979 18755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:52.252830 18835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:52.254416 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 18755
I20260812 02:29:52.254662 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:52.255338 12323 external_mini_cluster.cc:1428] Setting key 0324944680fccda2483a37359be5ae88
I20260812 02:29:52.266261 18835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822: Bootstrap starting.
I20260812 02:29:52.270076 18835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:52.271029 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:52.257310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.12.8.254:46711)
0812 02:29:52.258404 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:52.258423 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:52.258474 (+ 51us) server_negotiation.cc:407] Connection header received
0812 02:29:52.260801 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:52.260830 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:52.261437 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:52.261855 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:52.262426 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:52.264418 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:52.265825 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:52.266859 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:52.267030 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:52.268176 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:52.268187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:52.268206 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:52.268907 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:52.269033 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:52.269061 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:52.269202 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:52.269434 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 02:29:52.269746 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":329,"threads_started":1}
I20260812 02:29:52.271396 18835 log.cc:824] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:52.274458 18835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822: No bootstrap required, opened a new log
I20260812 02:29:52.274924 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:29:52.281893 18835 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6f48b8b294e1896d7df9b22e01822" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46711 } }
I20260812 02:29:52.282306 18835 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:52.282407 18835 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e6f48b8b294e1896d7df9b22e01822, State: Initialized, Role: FOLLOWER
I20260812 02:29:52.282928 18835 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6f48b8b294e1896d7df9b22e01822" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46711 } }
I20260812 02:29:52.283140 18835 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:52.283263 18835 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:52.283414 18835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:52.285766 18835 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6f48b8b294e1896d7df9b22e01822" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46711 } }
I20260812 02:29:52.286260 18835 leader_election.cc:302] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2e6f48b8b294e1896d7df9b22e01822; no voters:
I20260812 02:29:52.286832 18835 leader_election.cc:288] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:52.286962 18841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:52.287636 18841 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [term 1 LEADER]: Becoming Leader. State: Replica: b2e6f48b8b294e1896d7df9b22e01822, State: Running, Role: LEADER
I20260812 02:29:52.288151 18841 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6f48b8b294e1896d7df9b22e01822" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46711 } }
I20260812 02:29:52.288770 18835 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:52.290963 18842 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2e6f48b8b294e1896d7df9b22e01822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6f48b8b294e1896d7df9b22e01822" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46711 } } }
I20260812 02:29:52.290879 18843 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2e6f48b8b294e1896d7df9b22e01822. Latest consensus state: current_term: 1 leader_uuid: "b2e6f48b8b294e1896d7df9b22e01822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6f48b8b294e1896d7df9b22e01822" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46711 } } }
I20260812 02:29:52.291455 18842 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:52.291502 18843 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:52.292395 18850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:52.298086 18850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:52.307233 18850 catalog_manager.cc:1347] Generated new cluster ID: 2ec8b582bd3f42aba63b8c69abf744f4
I20260812 02:29:52.307374 18850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:52.324122 18850 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:52.325961 18850 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:52.334570 18850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2e6f48b8b294e1896d7df9b22e01822: Generated new TSK 0
I20260812 02:29:52.335567 18850 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 02:29:52.374153 18873 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 02:29:52.374310 18873 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 02:29:52.375289 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:52.347569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.12.8.254:46711)
0812 02:29:52.347787 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:52.347793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:52.348206 (+ 413us) server_negotiation.cc:407] Connection header received
0812 02:29:52.348495 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:52.348500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:52.348595 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:52.348740 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:29:52.349344 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:52.350082 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:52.352110 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:52.352575 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:52.352683 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:52.353326 (+ 643us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:29:52.374680 (+ 21354us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 02:29:52.375069 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":90}
W20260812 02:29:52.375425 18838 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 02:29:52.375700 18872 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:29:52.347353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46711 (local address 127.0.0.1:56880)
0812 02:29:52.348058 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:52.348079 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:52.348270 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:52.348772 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:52.348778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:52.348792 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:29:52.349125 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:52.349138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:52.350232 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:52.350239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:52.351933 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:52.351948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:52.352757 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:52.352764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:52.352967 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:52.353016 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:29:52.374980 (+ 21964us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 02:29:52.375297 (+ 317us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.8.254:46711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":360,"threads_started":1}
I20260812 02:29:52.377308 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 18755
I20260812 02:29:52.378587 18830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:52.472944 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 18755
2026-08-12T02:29:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (761 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 02:29:52.532045 12323 mini_oidc.cc:152] Starting JWKS server
I20260812 02:29:52.534075 12323 webserver.cc:460] Webserver started at https://127.0.0.1:36033/ using document root <none> and password file <none>
I20260812 02:29:52.534278 12323 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:29:52.534914 12323 webserver.cc:460] Webserver started at http://127.0.0.1:34737/ using document root <none> and password file <none>
2026-08-12T02:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:52Z Disabled control of system clock
I20260812 02:29:52.575946 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:37531
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:37531
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36033/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 02:29:52.994927 18882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:52.995361 18882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:52.995488 18882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:53.004333 18882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:53.004521 18882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:53.004575 18882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:53.004683 18882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:53.004756 18882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:29:53.004868 18882 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:29:53.004945 18882 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36033/jwks.json
I20260812 02:29:53.016170 18882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:37531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:37531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36033/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:53.018100 18882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:53.020421 18882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:53.034827 18888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:53.034914 18889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:53.035259 18892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:53.035934 18882 server_base.cc:1034] running on GCE node
I20260812 02:29:53.036963 18882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:53.038887 18882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:53.040115 18882 hybrid_clock.cc:648] HybridClock initialized: now 1786501793040045 us; error 66 us; skew 500 ppm
I20260812 02:29:53.040557 18882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:29:53.043602 18882 webserver.cc:460] Webserver started at http://127.12.8.254:43401/ using document root <none> and password file <none>
I20260812 02:29:53.044485 18882 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:53.044591 18882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:53.044981 18882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:53.048625 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "4a8c84abaef14ec9b57392a3081db469"
format_stamp: "Formatted at 2026-08-12 02:29:53 on dist-test-slave-vq2q"
server_key: "7603c5e4d4dc29530e7f90555edc7b8d"
server_key_iv: "6646a436916a892462ac228b983e4024"
server_key_version: "encryptionkey@0"
I20260812 02:29:53.049683 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "4a8c84abaef14ec9b57392a3081db469"
format_stamp: "Formatted at 2026-08-12 02:29:53 on dist-test-slave-vq2q"
server_key: "7603c5e4d4dc29530e7f90555edc7b8d"
server_key_iv: "6646a436916a892462ac228b983e4024"
server_key_version: "encryptionkey@0"
I20260812 02:29:53.057334 18882 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 02:29:53.062161 18899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:53.064110 18882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:53.064407 18882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "4a8c84abaef14ec9b57392a3081db469"
format_stamp: "Formatted at 2026-08-12 02:29:53 on dist-test-slave-vq2q"
server_key: "7603c5e4d4dc29530e7f90555edc7b8d"
server_key_iv: "6646a436916a892462ac228b983e4024"
server_key_version: "encryptionkey@0"
I20260812 02:29:53.064692 18882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:53.118047 18882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:53.119249 18882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:53.119652 18882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:53.142592 18882 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:37531
I20260812 02:29:53.142639 18964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:37531 every 8 connection(s)
I20260812 02:29:53.144501 18882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:53.149336 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 18882
I20260812 02:29:53.149580 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:53.149595 18965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:53.150486 12323 external_mini_cluster.cc:1428] Setting key 5c29efcefef603792455ba7f74f651a7
I20260812 02:29:53.165040 18965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469: Bootstrap starting.
I20260812 02:29:53.168031 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:53.153226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.12.8.254:37531)
0812 02:29:53.154680 (+ 1454us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:53.154697 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:53.154762 (+ 65us) server_negotiation.cc:407] Connection header received
0812 02:29:53.156766 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:53.156796 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:53.157517 (+ 721us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:53.157978 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:53.158734 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:53.160224 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:53.162232 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:53.163139 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:53.163305 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:53.164692 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:53.164701 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:53.164719 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:53.165520 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:53.165653 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:53.165691 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:53.165812 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:53.166221 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 02:29:53.166663 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":471,"threads_started":1}
I20260812 02:29:53.168843 18965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:53.170120 18965 log.cc:824] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:53.173102 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:29:53.174016 18965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469: No bootstrap required, opened a new log
I20260812 02:29:53.183000 18965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8c84abaef14ec9b57392a3081db469" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37531 } }
I20260812 02:29:53.183557 18965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:53.183671 18965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8c84abaef14ec9b57392a3081db469, State: Initialized, Role: FOLLOWER
I20260812 02:29:53.184408 18965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8c84abaef14ec9b57392a3081db469" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37531 } }
I20260812 02:29:53.184667 18965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:53.184819 18965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:53.185014 18965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:53.187628 18965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8c84abaef14ec9b57392a3081db469" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37531 } }
I20260812 02:29:53.188349 18965 leader_election.cc:302] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a8c84abaef14ec9b57392a3081db469; no voters:
I20260812 02:29:53.189105 18965 leader_election.cc:288] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:53.189338 18971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:53.190246 18971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [term 1 LEADER]: Becoming Leader. State: Replica: 4a8c84abaef14ec9b57392a3081db469, State: Running, Role: LEADER
I20260812 02:29:53.190902 18971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8c84abaef14ec9b57392a3081db469" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37531 } }
I20260812 02:29:53.191680 18965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:53.194772 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a8c84abaef14ec9b57392a3081db469. Latest consensus state: current_term: 1 leader_uuid: "4a8c84abaef14ec9b57392a3081db469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8c84abaef14ec9b57392a3081db469" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37531 } } }
I20260812 02:29:53.195173 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:53.196214 18972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a8c84abaef14ec9b57392a3081db469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8c84abaef14ec9b57392a3081db469" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37531 } } }
I20260812 02:29:53.196853 18972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:53.205765 18980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:53.210973 18980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:53.220857 18980 catalog_manager.cc:1347] Generated new cluster ID: 8bdb79e3bcd14a869ecaefa05bd43363
I20260812 02:29:53.220999 18980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:53.235939 18980 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:53.238306 18980 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:53.250197 18980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a8c84abaef14ec9b57392a3081db469: Generated new TSK 0
I20260812 02:29:53.251699 18980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:29:53.291455 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:53.281401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.12.8.254:37531)
0812 02:29:53.281662 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:53.281670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:53.281944 (+ 274us) server_negotiation.cc:407] Connection header received
0812 02:29:53.282201 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:53.282209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:53.282340 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:53.282592 (+ 252us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:29:53.283100 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:53.284312 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:53.285907 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:53.286722 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:53.286886 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:53.287471 (+ 585us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:29:53.290568 (+ 3097us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36033/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 02:29:53.291029 (+ 461us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36033/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":100}
W20260812 02:29:53.291638 18969 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36033/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 02:29:53.292690 19001 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:29:53.281163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:37531 (local address 127.0.0.1:51440)
0812 02:29:53.281832 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:53.281849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:53.281978 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:53.282586 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:53.282589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:53.282599 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:29:53.282931 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:53.282939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:53.284502 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:53.284506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:53.285738 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:53.285750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:53.286936 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:53.286940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:53.287067 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:53.287103 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:29:53.290832 (+ 3729us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36033/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 02:29:53.292240 (+ 1408us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.8.254:37531: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36033/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":1073,"thread_start_us":348,"threads_started":1}
I20260812 02:29:53.295883 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 18882
I20260812 02:29:53.297801 18960 generic_service.cc:196] Checking for leaks (request via RPC)
E20260812 02:29:53.303906 19002 webserver.cc:533] Webserver: sslize failed:
E20260812 02:29:53.304028 19002 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 02:29:53.403216 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 18882
2026-08-12T02:29:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (931 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T02:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:53Z Disabled control of system clock
I20260812 02:29:53.507891 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:38971
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:38971
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/testchainca.cert with env {}
W20260812 02:29:53.874794 19008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:53.875121 19008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:53.875236 19008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:53.882668 19008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:53.882800 19008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:53.882844 19008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:53.882932 19008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:53.882997 19008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:29:53.883083 19008 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:29:53.883118 19008 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 02:29:53.893457 19008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:38971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:53.894883 19008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:53.896198 19008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:53.909003 19013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:53.908993 19017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:53.909155 19014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:53.911180 19008 server_base.cc:1034] running on GCE node
I20260812 02:29:53.912322 19008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:53.914484 19008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:53.915779 19008 hybrid_clock.cc:648] HybridClock initialized: now 1786501793915710 us; error 50 us; skew 500 ppm
I20260812 02:29:53.916348 19008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:29:53.919502 19008 webserver.cc:460] Webserver started at http://127.12.8.254:36299/ using document root <none> and password file <none>
I20260812 02:29:53.920455 19008 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:53.920575 19008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:53.920974 19008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:53.924934 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "de28be3e6e70406fb8e3e74eb26d5cf0"
format_stamp: "Formatted at 2026-08-12 02:29:53 on dist-test-slave-vq2q"
server_key: "fd26d69e1e3c05207b28963f0dee2285"
server_key_iv: "d4bba403c84a1bb213376ff076e03eef"
server_key_version: "encryptionkey@0"
I20260812 02:29:53.926122 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "de28be3e6e70406fb8e3e74eb26d5cf0"
format_stamp: "Formatted at 2026-08-12 02:29:53 on dist-test-slave-vq2q"
server_key: "fd26d69e1e3c05207b28963f0dee2285"
server_key_iv: "d4bba403c84a1bb213376ff076e03eef"
server_key_version: "encryptionkey@0"
I20260812 02:29:53.933452 19008 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 02:29:53.938227 19026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:53.940157 19008 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:29:53.940402 19008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "de28be3e6e70406fb8e3e74eb26d5cf0"
format_stamp: "Formatted at 2026-08-12 02:29:53 on dist-test-slave-vq2q"
server_key: "fd26d69e1e3c05207b28963f0dee2285"
server_key_iv: "d4bba403c84a1bb213376ff076e03eef"
server_key_version: "encryptionkey@0"
I20260812 02:29:53.940685 19008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:53.959299 19008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:53.960403 19008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:53.960711 19008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:53.982775 19085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:38971 every 8 connection(s)
I20260812 02:29:53.982728 19008 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:38971
I20260812 02:29:53.984742 19008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:53.987152 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 19008
I20260812 02:29:53.987423 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:53.988271 12323 external_mini_cluster.cc:1428] Setting key d70cfcb434162f0a5102bc1527c408af
I20260812 02:29:53.989226 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:54.004843 19086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0: Bootstrap starting.
I20260812 02:29:54.005889 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:53.990962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.12.8.254:38971)
0812 02:29:53.992324 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:53.992340 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:53.992414 (+ 74us) server_negotiation.cc:407] Connection header received
0812 02:29:53.994484 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:53.994515 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:53.995187 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:53.995682 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:53.996411 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:53.998444 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:54.000097 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:54.001143 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:54.001383 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:54.002775 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:54.002786 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:54.002814 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:29:54.003559 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:54.003676 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:29:54.003718 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:54.003869 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:54.004107 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 02:29:54.004467 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":443,"threads_started":1}
I20260812 02:29:54.008635 19086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:54.009843 19086 log.cc:824] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:29:54.011291 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:29:54.014147 19086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0: No bootstrap required, opened a new log
I20260812 02:29:54.024300 19086 raft_consensus.cc:348] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de28be3e6e70406fb8e3e74eb26d5cf0" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 38971 } }
I20260812 02:29:54.024979 19086 raft_consensus.cc:374] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:54.025120 19086 raft_consensus.cc:729] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de28be3e6e70406fb8e3e74eb26d5cf0, State: Initialized, Role: FOLLOWER
I20260812 02:29:54.026008 19086 consensus_queue.cc:260] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de28be3e6e70406fb8e3e74eb26d5cf0" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 38971 } }
I20260812 02:29:54.026283 19086 raft_consensus.cc:388] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:54.026443 19086 raft_consensus.cc:482] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:54.026649 19086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:54.029309 19086 raft_consensus.cc:504] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de28be3e6e70406fb8e3e74eb26d5cf0" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 38971 } }
I20260812 02:29:54.030045 19086 leader_election.cc:302] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de28be3e6e70406fb8e3e74eb26d5cf0; no voters:
I20260812 02:29:54.030813 19086 leader_election.cc:288] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:54.031067 19092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:54.032374 19092 raft_consensus.cc:686] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [term 1 LEADER]: Becoming Leader. State: Replica: de28be3e6e70406fb8e3e74eb26d5cf0, State: Running, Role: LEADER
I20260812 02:29:54.033057 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de28be3e6e70406fb8e3e74eb26d5cf0" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 38971 } }
I20260812 02:29:54.033733 19086 sys_catalog.cc:564] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:54.036787 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de28be3e6e70406fb8e3e74eb26d5cf0. Latest consensus state: current_term: 1 leader_uuid: "de28be3e6e70406fb8e3e74eb26d5cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de28be3e6e70406fb8e3e74eb26d5cf0" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 38971 } } }
I20260812 02:29:54.037305 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:54.037679 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de28be3e6e70406fb8e3e74eb26d5cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de28be3e6e70406fb8e3e74eb26d5cf0" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 38971 } } }
I20260812 02:29:54.038028 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:54.042418 19099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:54.047654 19099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:54.055949 19099 catalog_manager.cc:1347] Generated new cluster ID: 16cdcfe9b6444e8dad289100e72dc1e6
I20260812 02:29:54.056105 19099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:54.081005 19099 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:54.082511 19099 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:54.090348 19099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de28be3e6e70406fb8e3e74eb26d5cf0: Generated new TSK 0
I20260812 02:29:54.091221 19099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:29:54.187418 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:54.119736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.12.8.254:38971)
0812 02:29:54.119992 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:54.119999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:54.120328 (+ 329us) server_negotiation.cc:407] Connection header received
0812 02:29:54.120637 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:54.120644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:54.120770 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:54.121033 (+ 263us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:29:54.121584 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:54.122842 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:54.124432 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:54.125248 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:54.125404 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:54.126136 (+ 732us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:29:54.186467 (+ 60331us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 02:29:54.187003 (+ 536us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47240: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":104}
W20260812 02:29:54.187616 19090 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47240: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 02:29:54.187584 19121 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:29:54.119501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:38971 (local address 127.0.0.1:47240)
0812 02:29:54.120209 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:54.120227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:54.120384 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:54.121059 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:54.121063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:54.121074 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:29:54.121411 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:54.121421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:54.123043 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:54.123048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:54.124257 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:54.124270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:54.125460 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:54.125465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:54.125633 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:54.125674 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:29:54.186901 (+ 61227us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 02:29:54.187191 (+ 290us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.8.254:38971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":382,"threads_started":1}
I20260812 02:29:54.189953 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 19008
I20260812 02:29:54.191326 19081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:29:54.295459 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 19008
2026-08-12T02:29:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (875 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (500 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1506 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:29:56 dist-test-slave-vq2q krb5kdc[19145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:29:56 dist-test-slave-vq2q krb5kdc[19145](info): set up 2 sockets
Aug 12 02:29:56 dist-test-slave-vq2q krb5kdc[19145](info): commencing operation
krb5kdc: starting...
W20260812 02:29:57.438426 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:29:57 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:29:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:29:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:29:57.753090 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43035
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:58.218864 19161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:58.219301 19161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:58.219411 19161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:58.227979 19161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:29:58.228149 19161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:58.228207 19161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:58.228271 19161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:29:58.228319 19161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:29:58.239881 19161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:58.241727 19161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:58.243525 19161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:58.257972 19167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:58.258045 19171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:58.258077 19166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:58.258713 19161 server_base.cc:1034] running on GCE node
I20260812 02:29:58.259801 19161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:58.261852 19161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:58.263129 19161 hybrid_clock.cc:648] HybridClock initialized: now 1786501798263067 us; error 55 us; skew 500 ppm
I20260812 02:29:58.263587 19161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:58 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501798, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:58.267671 19161 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:29:58.269948 19161 webserver.cc:460] Webserver started at http://127.12.8.254:43291/ using document root <none> and password file <none>
I20260812 02:29:58.270931 19161 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:58.271093 19161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:58.271521 19161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:58.275624 19161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "90e4be5180e04a6185317cbe9eab9212"
format_stamp: "Formatted at 2026-08-12 02:29:58 on dist-test-slave-vq2q"
server_key: "cf46807db6c8cbe5b90111a1fabe1e9a"
server_key_iv: "13be916a1babf3490989d23ecb6ca026"
server_key_version: "encryptionkey@0"
I20260812 02:29:58.276703 19161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "90e4be5180e04a6185317cbe9eab9212"
format_stamp: "Formatted at 2026-08-12 02:29:58 on dist-test-slave-vq2q"
server_key: "cf46807db6c8cbe5b90111a1fabe1e9a"
server_key_iv: "13be916a1babf3490989d23ecb6ca026"
server_key_version: "encryptionkey@0"
I20260812 02:29:58.284366 19161 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 02:29:58.289422 19180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:58.291393 19161 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:29:58.291656 19161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "90e4be5180e04a6185317cbe9eab9212"
format_stamp: "Formatted at 2026-08-12 02:29:58 on dist-test-slave-vq2q"
server_key: "cf46807db6c8cbe5b90111a1fabe1e9a"
server_key_iv: "13be916a1babf3490989d23ecb6ca026"
server_key_version: "encryptionkey@0"
I20260812 02:29:58.291922 19161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:58.318648 19161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:58.323663 19161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:58.323992 19161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:58.344389 19161 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:43035
I20260812 02:29:58.344432 19241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:43035 every 8 connection(s)
I20260812 02:29:58.346132 19161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:29:58.351420 19243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:29:58.355178 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 19161
I20260812 02:29:58.355448 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:29:58.356158 12323 external_mini_cluster.cc:1428] Setting key e56caa579ce2e1cf932b3b8bd09434b0
I20260812 02:29:58.367769 19243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212: Bootstrap starting.
I20260812 02:29:58.372649 19243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212: Neither blocks nor log segments found. Creating new log.
I20260812 02:29:58.374452 19243 log.cc:824] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:29:58 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:58.378795 19243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212: No bootstrap required, opened a new log
I20260812 02:29:58.386459 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:58.358915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.12.8.254:43035)
0812 02:29:58.359919 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:58.359938 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:58.359998 (+ 60us) server_negotiation.cc:407] Connection header received
0812 02:29:58.362404 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:58.362447 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:58.363285 (+ 838us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:58.363913 (+ 628us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:58.365753 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:58.367930 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:58.369401 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:58.370643 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:58.370865 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:58.376447 (+ 5582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:58.376459 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:58.376483 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:58.376539 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:58.380494 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:58.381753 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:58.381763 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:58.381772 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:58.382011 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:58.382558 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:58.382562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:58.382564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:58.383319 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:58.383609 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:58.384282 (+ 673us) server_negotiation.cc:299] Negotiation successful
0812 02:29:58.384900 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":423,"threads_started":1}
I20260812 02:29:58.389425 19243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e4be5180e04a6185317cbe9eab9212" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43035 } }
I20260812 02:29:58.389900 19243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:29:58.390008 19243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e4be5180e04a6185317cbe9eab9212, State: Initialized, Role: FOLLOWER
I20260812 02:29:58.390781 19243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e4be5180e04a6185317cbe9eab9212" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43035 } }
I20260812 02:29:58.391273 19243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:29:58.391443 19243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:29:58.391639 19243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:29:58.394667 19243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e4be5180e04a6185317cbe9eab9212" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43035 } }
I20260812 02:29:58.395424 19243 leader_election.cc:302] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e4be5180e04a6185317cbe9eab9212; no voters:
I20260812 02:29:58.396319 19243 leader_election.cc:288] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:29:58.396512 19249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:29:58.397351 19249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [term 1 LEADER]: Becoming Leader. State: Replica: 90e4be5180e04a6185317cbe9eab9212, State: Running, Role: LEADER
I20260812 02:29:58.398030 19249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e4be5180e04a6185317cbe9eab9212" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43035 } }
I20260812 02:29:58.398957 19243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:29:58.401926 19251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e4be5180e04a6185317cbe9eab9212. Latest consensus state: current_term: 1 leader_uuid: "90e4be5180e04a6185317cbe9eab9212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e4be5180e04a6185317cbe9eab9212" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43035 } } }
I20260812 02:29:58.402604 19251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:58.403720 19250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e4be5180e04a6185317cbe9eab9212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e4be5180e04a6185317cbe9eab9212" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43035 } } }
I20260812 02:29:58.404191 19250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212 [sys.catalog]: This master's current role is: LEADER
I20260812 02:29:58.404898 19256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:29:58.412348 19256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:29:58.424629 19256 catalog_manager.cc:1347] Generated new cluster ID: e2a50622c97e4f40a23836e0932e9787
I20260812 02:29:58.424782 19256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:29:58.442565 19256 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:29:58.444257 19256 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:29:58.452389 19256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e4be5180e04a6185317cbe9eab9212: Generated new TSK 0
I20260812 02:29:58.453845 19256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:29:58.533273 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43035
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:58.949446 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:58.949755 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:58.949880 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:58.957726 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:58.957844 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:58.957993 19270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:29:58.968093 19270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:58.969552 19270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:58.970954 19270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:58.984665 19277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:58.984692 19280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:58.985234 19276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:58.985677 19270 server_base.cc:1034] running on GCE node
I20260812 02:29:58.986374 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:58.987370 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:58.988570 19270 hybrid_clock.cc:648] HybridClock initialized: now 1786501798988517 us; error 49 us; skew 500 ppm
I20260812 02:29:58.988894 19270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:58 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501798, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:58.992121 19270 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:29:58.993654 19270 webserver.cc:460] Webserver started at http://127.12.8.193:34455/ using document root <none> and password file <none>
I20260812 02:29:58.994371 19270 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:58.994491 19270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:58.994807 19270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:58.998077 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "c20d45477ee2488180ac71a4a9cc4322"
format_stamp: "Formatted at 2026-08-12 02:29:58 on dist-test-slave-vq2q"
server_key: "927d60ae4a2a622ab45ef9ca9cb1b9af"
server_key_iv: "2c7d5de262125b0b7f8d3cb73314c400"
server_key_version: "encryptionkey@0"
I20260812 02:29:58.998939 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "c20d45477ee2488180ac71a4a9cc4322"
format_stamp: "Formatted at 2026-08-12 02:29:58 on dist-test-slave-vq2q"
server_key: "927d60ae4a2a622ab45ef9ca9cb1b9af"
server_key_iv: "2c7d5de262125b0b7f8d3cb73314c400"
server_key_version: "encryptionkey@0"
I20260812 02:29:59.004544 19270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260812 02:29:59.008394 19289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:59.009932 19270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:29:59.010097 19270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "c20d45477ee2488180ac71a4a9cc4322"
format_stamp: "Formatted at 2026-08-12 02:29:58 on dist-test-slave-vq2q"
server_key: "927d60ae4a2a622ab45ef9ca9cb1b9af"
server_key_iv: "2c7d5de262125b0b7f8d3cb73314c400"
server_key_version: "encryptionkey@0"
I20260812 02:29:59.010305 19270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:59.037081 19270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:59.043059 19270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:59.043488 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:59.044960 19270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:59.047122 19270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:59.047245 19270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:59.047387 19270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:59.047446 19270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:59.090081 19270 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:38433
I20260812 02:29:59.090214 19420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:38433 every 8 connection(s)
I20260812 02:29:59.091585 19270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:29:59.097987 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 19270
I20260812 02:29:59.098232 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:29:59.099059 12323 external_mini_cluster.cc:1428] Setting key b8574a84600048009e74d3e0b69b9385
Aug 12 02:29:59 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501798, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:59.119122 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:59.094624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:56395 (local address 127.12.8.254:43035)
0812 02:29:59.095205 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:59.095211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:59.096279 (+ 1068us) server_negotiation.cc:407] Connection header received
0812 02:29:59.098476 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:59.098483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:59.098596 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:59.098703 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:59.100975 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:59.101830 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:59.103537 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:59.103966 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:59.104087 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:59.110344 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:59.110353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:59.110356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:59.110415 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:59.113992 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:59.115156 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:59.115162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:59.115164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:59.115477 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:59.116031 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:59.116035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:59.116037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:59.116684 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:59.116971 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:59.118379 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0812 02:29:59.118700 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":326,"threads_started":1}
I20260812 02:29:59.120366 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:59.095180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43035 (local address 127.12.8.193:56395)
0812 02:29:59.096073 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:59.096125 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:59.098121 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:59.099005 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:59.099014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:59.100025 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:59.100779 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:59.100802 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:59.102004 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:59.102008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:59.103358 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:59.103374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:59.104228 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:59.104248 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:59.104500 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:59.105363 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:59.105392 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:59.109827 (+ 4435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:59.114237 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:59.114250 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:59.114274 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:59.114999 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:59.115636 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:59.115642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:59.115645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:59.115897 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:59.117028 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:59.117041 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:59.117818 (+ 777us) client_negotiation.cc:769] Sending connection context
0812 02:29:59.118184 (+ 366us) client_negotiation.cc:240] Negotiation successful
0812 02:29:59.118933 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":379,"threads_started":1}
I20260812 02:29:59.122345 19421 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43035
I20260812 02:29:59.122740 19421 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:59.123752 19421 heartbeater.cc:511] Master 127.12.8.254:43035 requested a full tablet report, sending...
I20260812 02:29:59.127636 19200 ts_manager.cc:194] Registered new tserver with Master: c20d45477ee2488180ac71a4a9cc4322 (127.12.8.193:38433)
I20260812 02:29:59.130059 19200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:56395
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:29:59.166234 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43035
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:29:59.522641 19427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:29:59.522909 19427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:29:59.523029 19427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:29:59.530344 19427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:29:59.530462 19427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:29:59.530620 19427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:29:59.540562 19427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:29:59.541917 19427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:29:59.543237 19427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:29:59.556752 19433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:59.556726 19437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:29:59.556821 19434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:29:59.557533 19427 server_base.cc:1034] running on GCE node
I20260812 02:29:59.558172 19427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:29:59.559098 19427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:29:59.560325 19427 hybrid_clock.cc:648] HybridClock initialized: now 1786501799560284 us; error 31 us; skew 500 ppm
I20260812 02:29:59.560675 19427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:29:59 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501799, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:29:59.564131 19427 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:29:59.565773 19427 webserver.cc:460] Webserver started at http://127.12.8.194:39849/ using document root <none> and password file <none>
I20260812 02:29:59.566468 19427 fs_manager.cc:359] Metadata directory not provided
I20260812 02:29:59.566586 19427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:29:59.566908 19427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:29:59.569979 19427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "c23059f85e084d688da7ceaad26ad10f"
format_stamp: "Formatted at 2026-08-12 02:29:59 on dist-test-slave-vq2q"
server_key: "3ec7bb078456e499b97e9c852a35d4ec"
server_key_iv: "3658184ee1e273778d7342d1880dd7b6"
server_key_version: "encryptionkey@0"
I20260812 02:29:59.570752 19427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "c23059f85e084d688da7ceaad26ad10f"
format_stamp: "Formatted at 2026-08-12 02:29:59 on dist-test-slave-vq2q"
server_key: "3ec7bb078456e499b97e9c852a35d4ec"
server_key_iv: "3658184ee1e273778d7342d1880dd7b6"
server_key_version: "encryptionkey@0"
I20260812 02:29:59.576023 19427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:29:59.579463 19445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:59.580792 19427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 02:29:59.580938 19427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "c23059f85e084d688da7ceaad26ad10f"
format_stamp: "Formatted at 2026-08-12 02:29:59 on dist-test-slave-vq2q"
server_key: "3ec7bb078456e499b97e9c852a35d4ec"
server_key_iv: "3658184ee1e273778d7342d1880dd7b6"
server_key_version: "encryptionkey@0"
I20260812 02:29:59.581113 19427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:29:59.598954 19427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:29:59.603221 19427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:29:59.603478 19427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:29:59.604435 19427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:29:59.605871 19427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:29:59.605962 19427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:59.606058 19427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:29:59.606102 19427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:29:59.636965 19427 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:33961
I20260812 02:29:59.637036 19582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:33961 every 8 connection(s)
I20260812 02:29:59.638249 19427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:29:59.643199 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 19427
I20260812 02:29:59.643400 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:29:59.643819 12323 external_mini_cluster.cc:1428] Setting key 14ed912dae7cceb39354b6af001ffec6
Aug 12 02:29:59 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501799, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:29:59.660454 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:59.640793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:44301 (local address 127.12.8.254:43035)
0812 02:29:59.641343 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0812 02:29:59.641351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:29:59.642412 (+ 1061us) server_negotiation.cc:407] Connection header received
0812 02:29:59.644414 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:29:59.644420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:29:59.644520 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:29:59.644633 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:29:59.646724 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:59.647521 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:59.649672 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:29:59.650108 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:29:59.650198 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:59.654160 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:29:59.654167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:29:59.654172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:29:59.654213 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:29:59.657152 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:59.658097 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:59.658101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:59.658104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:59.658229 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:29:59.658742 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:29:59.658746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:29:59.658748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:29:59.659032 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:29:59.659150 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:29:59.659978 (+ 828us) server_negotiation.cc:299] Negotiation successful
0812 02:29:59.660153 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":244,"threads_started":1}
I20260812 02:29:59.661835 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:29:59.641310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43035 (local address 127.12.8.194:44301)
0812 02:29:59.642224 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0812 02:29:59.642270 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:29:59.644126 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:29:59.644904 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:29:59.644915 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:29:59.645766 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:29:59.646527 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:59.646556 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:59.647688 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:59.647694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:59.649512 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:29:59.649529 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:29:59.650340 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:29:59.650347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:29:59.650610 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:29:59.651296 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:29:59.651337 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:29:59.653903 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:29:59.657338 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:59.657347 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:59.657365 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:59.657962 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:59.658345 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:29:59.658350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:29:59.658352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:29:59.658618 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:29:59.659154 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:29:59.659163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:29:59.659603 (+ 440us) client_negotiation.cc:769] Sending connection context
0812 02:29:59.659900 (+ 297us) client_negotiation.cc:240] Negotiation successful
0812 02:29:59.660437 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":354,"threads_started":1}
I20260812 02:29:59.663256 19584 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43035
I20260812 02:29:59.663579 19584 heartbeater.cc:464] Registering TS with master...
I20260812 02:29:59.664299 19584 heartbeater.cc:511] Master 127.12.8.254:43035 requested a full tablet report, sending...
I20260812 02:29:59.665714 19200 ts_manager.cc:194] Registered new tserver with Master: c23059f85e084d688da7ceaad26ad10f (127.12.8.194:33961)
I20260812 02:29:59.666733 19200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:44301
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:29:59.704514 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43035
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:00.060851 19590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:00.061165 19590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:00.061333 19590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:00.068957 19590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:00.069103 19590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:00.069283 19590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:00.079735 19590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:00.081084 19590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:00.082804 19590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:00.100196 19600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:00.100538 19598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:00.100852 19597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:00.101217 19590 server_base.cc:1034] running on GCE node
I20260812 02:30:00.101961 19590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:00.103125 19590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:00.104362 19590 hybrid_clock.cc:648] HybridClock initialized: now 1786501800104284 us; error 72 us; skew 500 ppm
I20260812 02:30:00.104715 19590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:00.108327 19590 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:30:00.110169 19590 webserver.cc:460] Webserver started at http://127.12.8.195:39679/ using document root <none> and password file <none>
I20260812 02:30:00.110918 19590 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:00.111069 19590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:00.111398 19590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:00.114734 19590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "d89943bb28f64ab0a341bb5e2bb904f7"
format_stamp: "Formatted at 2026-08-12 02:30:00 on dist-test-slave-vq2q"
server_key: "8161805b40dab79282d805eafe76b31b"
server_key_iv: "de15d6a659cdc6065a79648d04e1516a"
server_key_version: "encryptionkey@0"
I20260812 02:30:00.115585 19590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "d89943bb28f64ab0a341bb5e2bb904f7"
format_stamp: "Formatted at 2026-08-12 02:30:00 on dist-test-slave-vq2q"
server_key: "8161805b40dab79282d805eafe76b31b"
server_key_iv: "de15d6a659cdc6065a79648d04e1516a"
server_key_version: "encryptionkey@0"
I20260812 02:30:00.121975 19590 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 02:30:00.126575 19609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:00.128391 19590 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:30:00.128607 19590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "d89943bb28f64ab0a341bb5e2bb904f7"
format_stamp: "Formatted at 2026-08-12 02:30:00 on dist-test-slave-vq2q"
server_key: "8161805b40dab79282d805eafe76b31b"
server_key_iv: "de15d6a659cdc6065a79648d04e1516a"
server_key_version: "encryptionkey@0"
I20260812 02:30:00.128856 19590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:00.135088 19421 heartbeater.cc:503] Master 127.12.8.254:43035 was elected leader, sending a full tablet report...
I20260812 02:30:00.166050 19590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:00.170825 19590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:00.171144 19590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:00.172374 19590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:00.174064 19590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:00.174146 19590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:00.174263 19590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:00.174324 19590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:00.212240 19590 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:33073
I20260812 02:30:00.212397 19739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:33073 every 8 connection(s)
I20260812 02:30:00.213697 19590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:00.216027 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 19590
I20260812 02:30:00.216239 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:30:00.216699 12323 external_mini_cluster.cc:1428] Setting key ab4baa716af09db8a8f22fc0d45c9931
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:00.241449 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:00.217074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:39541 (local address 127.12.8.254:43035)
0812 02:30:00.221237 (+ 4163us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.221245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.221276 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:30:00.222675 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.222680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.222834 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.222961 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.224928 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.226102 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.227891 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.228413 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.228557 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:00.233129 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:00.233136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:00.233140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:00.233232 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:00.236411 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.237838 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.237844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.237847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.238008 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.238764 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.238770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.238772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.239179 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:00.239380 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:00.240487 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0812 02:30:00.240838 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3951,"thread_start_us":3761,"threads_started":1}
I20260812 02:30:00.242774 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:00.218337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43035 (local address 127.12.8.195:39541)
0812 02:30:00.219619 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:00.219674 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:00.222227 (+ 2553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:00.223310 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:00.223323 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:00.223987 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:00.224704 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:00.224726 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.226349 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.226354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:00.227684 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:00.227700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.228704 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.228710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:00.228970 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:00.229718 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:00.229784 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:00.232774 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:00.236643 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:00.236653 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:00.236674 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:00.237675 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:00.238182 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:00.238188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:00.238191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:00.238593 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:00.239349 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:00.239360 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:00.239881 (+ 521us) client_negotiation.cc:769] Sending connection context
0812 02:30:00.240313 (+ 432us) client_negotiation.cc:240] Negotiation successful
0812 02:30:00.241088 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"thread_start_us":586,"threads_started":1}
I20260812 02:30:00.244988 19740 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43035
I20260812 02:30:00.245461 19740 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:00.246392 19740 heartbeater.cc:511] Master 127.12.8.254:43035 requested a full tablet report, sending...
I20260812 02:30:00.249002 19200 ts_manager.cc:194] Registered new tserver with Master: d89943bb28f64ab0a341bb5e2bb904f7 (127.12.8.195:33073)
I20260812 02:30:00.250249 19200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:39541
I20260812 02:30:00.252007 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/test-admin.keytab.
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:00.292047 12323 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 02:30:00.299045 19749 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:00.295684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.12.8.193:38433)
0812 02:30:00.296506 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.296514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.296548 (+ 34us) server_negotiation.cc:407] Connection header received
0812 02:30:00.296622 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.296629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.296955 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.297108 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.298596 (+ 1488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41098: BlockingRecv error: recv got EOF from 127.0.0.1:41098 (error 108)
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":393,"threads_started":1}
W20260812 02:30:00.299156 12323 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5cc' of type FILE
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:00.301748 12323 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:00.303529 12323 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:30:00.324242 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:00.305966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.12.8.193:38433)
0812 02:30:00.306249 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.306257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.306429 (+ 172us) server_negotiation.cc:407] Connection header received
0812 02:30:00.306629 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.306634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.306764 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.306870 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.308235 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.309395 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.310834 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.311460 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.311575 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:00.316894 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:00.316913 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:00.316930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:00.316990 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:00.320343 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.321553 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.321563 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.321566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.321740 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.322297 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.322300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.322303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.322779 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:00.323048 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:00.323609 (+ 561us) server_negotiation.cc:299] Negotiation successful
0812 02:30:00.323858 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260812 02:30:00.330988 19749 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:00.328787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.12.8.193:38433)
0812 02:30:00.329080 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.329087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.329446 (+ 359us) server_negotiation.cc:407] Connection header received
0812 02:30:00.329628 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.329632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.329697 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.329779 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.330676 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41116: BlockingRecv error: recv got EOF from 127.0.0.1:41116 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260812 02:30:00.331216 12323 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5cc' of type FILE
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:00.333149 12323 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:00.334826 12323 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:30:00.353081 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:00.336624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.12.8.193:38433)
0812 02:30:00.336945 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.336955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.337318 (+ 363us) server_negotiation.cc:407] Connection header received
0812 02:30:00.337542 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.337547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.337656 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.337756 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.338985 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.339789 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.341083 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.341641 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.341762 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:00.346505 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:00.346513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:00.346517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:00.346569 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:00.350011 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.350910 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.350914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.350917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.351083 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.351501 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.351505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.351507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.351852 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:00.352172 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:00.352487 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 02:30:00.352759 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260812 02:30:00.358831 19749 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:00.356764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.12.8.193:38433)
0812 02:30:00.357042 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.357048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.357337 (+ 289us) server_negotiation.cc:407] Connection header received
0812 02:30:00.357525 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.357529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.357620 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.357719 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.358544 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41132: BlockingRecv error: recv got EOF from 127.0.0.1:41132 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260812 02:30:00.359001 12323 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501741404412-12323-0/krb5kdc/krb5cc' of type FILE
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:00.360658 12323 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:00.362004 12323 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:30:00.378170 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:00.363630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.12.8.193:38433)
0812 02:30:00.363994 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.364008 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.364182 (+ 174us) server_negotiation.cc:407] Connection header received
0812 02:30:00.364398 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.364402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.364490 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.364589 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.365694 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.366476 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.367732 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.368152 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.368265 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:00.372396 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:00.372402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:00.372405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:00.372445 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:00.375458 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.376129 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.376134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.376136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.376310 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.376659 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.376662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.376664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.376974 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:00.377333 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:00.377575 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 02:30:00.377823 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260812 02:30:00.380061 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 19270
I20260812 02:30:00.394214 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:00.380846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.12.8.193:38433)
0812 02:30:00.381053 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.381060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.381523 (+ 463us) server_negotiation.cc:407] Connection header received
0812 02:30:00.381666 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.381669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.381733 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.381830 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.382987 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.383691 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.385166 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.385693 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.385856 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:00.388471 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:00.388477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:00.388481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:00.388521 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:00.391192 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.392027 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.392031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.392034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.392181 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.392682 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.392686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.392688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.392989 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:00.393243 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:00.393650 (+ 407us) server_negotiation.cc:299] Negotiation successful
0812 02:30:00.393838 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 02:30:00.395112 19416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:00.491968 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 19270
I20260812 02:30:00.520843 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 19427
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:30:00.540243 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:00.522055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.12.8.194:33961)
0812 02:30:00.522699 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.522708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.522742 (+ 34us) server_negotiation.cc:407] Connection header received
0812 02:30:00.522830 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.522836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.523084 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.523242 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.524540 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.525780 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.527456 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.528074 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.528189 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:00.533395 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:00.533414 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:00.533438 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:00.533484 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:00.536756 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.537802 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.537810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.537814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.537966 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.538421 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.538425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.538427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.538836 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:00.539109 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:00.539598 (+ 489us) server_negotiation.cc:299] Negotiation successful
0812 02:30:00.539900 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":342,"threads_started":1}
I20260812 02:30:00.542106 19578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:00.636909 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 19427
I20260812 02:30:00.661536 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 19590
Aug 12 02:30:00 dist-test-slave-vq2q krb5kdc[19145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:30:00.679952 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:00.662708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.12.8.195:33073)
0812 02:30:00.663444 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:00.663453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:00.663491 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:30:00.663566 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:00.663575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:00.663858 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:00.664040 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:00.665247 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.666347 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.667650 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:00.668304 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:00.668429 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:00.673241 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:00.673265 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:00.673292 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:00.673362 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:00.676508 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.677493 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.677503 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.677506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.677676 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:00.678098 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:00.678103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:00.678106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:00.678583 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:00.678924 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:00.679256 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 02:30:00.679584 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":333,"threads_started":1}
I20260812 02:30:00.681694 19734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:00.773542 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 19590
I20260812 02:30:00.797047 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 19161
I20260812 02:30:00.798444 19236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:00.925683 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 19161
2026-08-12T02:30:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4629 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:30:01 dist-test-slave-vq2q krb5kdc[19772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:30:01 dist-test-slave-vq2q krb5kdc[19772](info): set up 2 sockets
Aug 12 02:30:01 dist-test-slave-vq2q krb5kdc[19772](info): commencing operation
krb5kdc: starting...
W20260812 02:30:02.098084 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.002s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:30:02 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:30:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:30:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.254.keytab.
Entry for principal oryx/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.254.keytab.
I20260812 02:30:02.421916 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:35939
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.254.keytab
--principal=oryx/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:35939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:02.766717 19787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:02.767081 19787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:02.767202 19787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:02.774524 19787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:02.774641 19787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:02.774691 19787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:02.774784 19787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:02.774839 19787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:02.784823 19787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:35939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.254.keytab
--principal=oryx/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:35939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:02.786258 19787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:02.787632 19787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:02.799314 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:02.799336 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:02.799311 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:02.800163 19787 server_base.cc:1034] running on GCE node
I20260812 02:30:02.800966 19787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:02.802424 19787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:02.803663 19787 hybrid_clock.cc:648] HybridClock initialized: now 1786501802803605 us; error 41 us; skew 500 ppm
I20260812 02:30:02.804030 19787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:02 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:02.811116 19787 init.cc:377] Logged in from keytab as oryx/127.12.8.254@KRBTEST.COM (short username oryx)
I20260812 02:30:02.812716 19787 webserver.cc:460] Webserver started at http://127.12.8.254:37649/ using document root <none> and password file <none>
I20260812 02:30:02.813498 19787 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:02.813606 19787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:02.813882 19787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:02.816954 19787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "c787dfba77e6402dadf24679cfedd84b"
format_stamp: "Formatted at 2026-08-12 02:30:02 on dist-test-slave-vq2q"
server_key: "41053dd671b1a614f768261798a4db2a"
server_key_iv: "ffe915aab6dc31e4d543167ccb770a52"
server_key_version: "encryptionkey@0"
I20260812 02:30:02.817807 19787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "c787dfba77e6402dadf24679cfedd84b"
format_stamp: "Formatted at 2026-08-12 02:30:02 on dist-test-slave-vq2q"
server_key: "41053dd671b1a614f768261798a4db2a"
server_key_iv: "ffe915aab6dc31e4d543167ccb770a52"
server_key_version: "encryptionkey@0"
I20260812 02:30:02.823205 19787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:30:02.826930 19807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:02.828348 19787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:30:02.828537 19787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "c787dfba77e6402dadf24679cfedd84b"
format_stamp: "Formatted at 2026-08-12 02:30:02 on dist-test-slave-vq2q"
server_key: "41053dd671b1a614f768261798a4db2a"
server_key_iv: "ffe915aab6dc31e4d543167ccb770a52"
server_key_version: "encryptionkey@0"
I20260812 02:30:02.828725 19787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:02.844662 19787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:02.848966 19787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:02.849282 19787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:02.867959 19787 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:35939
I20260812 02:30:02.868001 19867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:35939 every 8 connection(s)
I20260812 02:30:02.869458 19787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:30:02.873246 19869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:02.875463 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 19787
I20260812 02:30:02.875713 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:30:02.876531 12323 external_mini_cluster.cc:1428] Setting key 6b2f17fc5b9b8c3edd420c3db28ef100
I20260812 02:30:02.885804 19869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: Bootstrap starting.
I20260812 02:30:02.889662 19869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:02.890934 19869 log.cc:824] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:30:02 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:02.894649 19869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: No bootstrap required, opened a new log
I20260812 02:30:02.901013 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:02.878714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.12.8.254:35939)
0812 02:30:02.879543 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:02.879561 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:02.879603 (+ 42us) server_negotiation.cc:407] Connection header received
0812 02:30:02.881319 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:02.881341 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:02.881704 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:02.882031 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:02.883801 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:02.885095 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:02.886799 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:02.887462 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:02.887586 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:02.892446 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:02.892459 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:02.892479 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:02.892524 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:02.896323 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:02.897045 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:02.897057 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:02.897067 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:02.897316 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:02.897703 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:02.897708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:02.897712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:02.898470 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:02.898796 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:02.899130 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 02:30:02.899578 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":367,"threads_started":1}
I20260812 02:30:02.904840 19869 raft_consensus.cc:348] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } }
I20260812 02:30:02.905181 19869 raft_consensus.cc:374] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:02.905294 19869 raft_consensus.cc:729] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c787dfba77e6402dadf24679cfedd84b, State: Initialized, Role: FOLLOWER
I20260812 02:30:02.905838 19869 consensus_queue.cc:260] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } }
I20260812 02:30:02.906050 19869 raft_consensus.cc:388] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:30:02.906148 19869 raft_consensus.cc:482] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:30:02.906270 19869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:02.908406 19869 raft_consensus.cc:504] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } }
I20260812 02:30:02.908916 19869 leader_election.cc:302] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c787dfba77e6402dadf24679cfedd84b; no voters:
I20260812 02:30:02.909637 19869 leader_election.cc:288] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:30:02.909898 19874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:02.910579 19874 raft_consensus.cc:686] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 1 LEADER]: Becoming Leader. State: Replica: c787dfba77e6402dadf24679cfedd84b, State: Running, Role: LEADER
I20260812 02:30:02.911108 19874 consensus_queue.cc:237] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } }
I20260812 02:30:02.911690 19869 sys_catalog.cc:564] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:02.913874 19876 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c787dfba77e6402dadf24679cfedd84b. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } }
I20260812 02:30:02.914309 19876 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:02.914892 19875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } }
I20260812 02:30:02.915223 19875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:02.915241 19879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:30:02.921139 19879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:30:02.931282 19879 catalog_manager.cc:1347] Generated new cluster ID: f2332070953147fab8eb91c78e45377b
I20260812 02:30:02.931419 19879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:30:02.952386 19879 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:30:02.954341 19879 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:30:02.961248 19879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: Generated new TSK 0
I20260812 02:30:02.962047 19879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab.
Entry for principal oryx/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab.
I20260812 02:30:03.046295 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:35939
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab
--principal=oryx/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:03.419677 19902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:03.419965 19902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:03.420063 19902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:03.427765 19902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:03.427899 19902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:03.428040 19902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:30:03.438438 19902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab
--principal=oryx/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:35939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:03.439779 19902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:03.441128 19902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:03.454932 19910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:03.455011 19908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:03.454962 19907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:03.462127 19902 server_base.cc:1034] running on GCE node
I20260812 02:30:03.462857 19902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:03.463923 19902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:03.465147 19902 hybrid_clock.cc:648] HybridClock initialized: now 1786501803465096 us; error 44 us; skew 500 ppm
I20260812 02:30:03.465512 19902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:03 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501803, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:03.468956 19902 init.cc:377] Logged in from keytab as oryx/127.12.8.193@KRBTEST.COM (short username oryx)
I20260812 02:30:03.470667 19902 webserver.cc:460] Webserver started at http://127.12.8.193:38951/ using document root <none> and password file <none>
I20260812 02:30:03.471412 19902 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:03.471521 19902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:03.471804 19902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:03.475014 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "572e2683344e422d87faad60b0bcb6d3"
format_stamp: "Formatted at 2026-08-12 02:30:03 on dist-test-slave-vq2q"
server_key: "714cc5b68ad11b36d729ad4263507231"
server_key_iv: "d388bcee997dbe5e3556a6ba045f3b0b"
server_key_version: "encryptionkey@0"
I20260812 02:30:03.475841 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "572e2683344e422d87faad60b0bcb6d3"
format_stamp: "Formatted at 2026-08-12 02:30:03 on dist-test-slave-vq2q"
server_key: "714cc5b68ad11b36d729ad4263507231"
server_key_iv: "d388bcee997dbe5e3556a6ba045f3b0b"
server_key_version: "encryptionkey@0"
I20260812 02:30:03.481683 19902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 02:30:03.485301 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:03.486683 19902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:30:03.486835 19902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "572e2683344e422d87faad60b0bcb6d3"
format_stamp: "Formatted at 2026-08-12 02:30:03 on dist-test-slave-vq2q"
server_key: "714cc5b68ad11b36d729ad4263507231"
server_key_iv: "d388bcee997dbe5e3556a6ba045f3b0b"
server_key_version: "encryptionkey@0"
I20260812 02:30:03.487058 19902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:03.515949 19902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:03.520612 19902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:03.520916 19902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:03.521999 19902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:03.523430 19902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:03.523504 19902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:03.523618 19902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:03.523670 19902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:03.556038 19902 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:45959
I20260812 02:30:03.556115 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:45959 every 8 connection(s)
I20260812 02:30:03.557399 19902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:30:03.559494 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 19902
I20260812 02:30:03.559741 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:30:03.560565 12323 external_mini_cluster.cc:1428] Setting key 5b66ef9ca0fb311cfd038768497a581b
Aug 12 02:30:03 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501803, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:03.577476 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:03.560069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:52913 (local address 127.12.8.254:35939)
0812 02:30:03.560507 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:03.560512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:03.561825 (+ 1313us) server_negotiation.cc:407] Connection header received
0812 02:30:03.563802 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:03.563808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:03.563887 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:03.563981 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:03.565535 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:03.566272 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:03.567812 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:03.568206 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:03.568313 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:03.571697 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:03.571703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:03.571706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:03.571740 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:03.574285 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:03.575006 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:03.575010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:03.575012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:03.575137 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:03.575554 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:03.575558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:03.575560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:03.575847 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:03.575994 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:03.576927 (+ 933us) server_negotiation.cc:299] Negotiation successful
0812 02:30:03.577155 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260812 02:30:03.578748 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:03.560738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:52913)
0812 02:30:03.561663 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:03.561712 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:03.563530 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:03.564222 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:03.564230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:03.564745 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:03.565372 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:03.565392 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:03.566427 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:03.566431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:03.567657 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:03.567674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:03.568425 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:03.568432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:03.568641 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:03.569307 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:03.569331 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:03.571485 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:03.574418 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:03.574427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:03.574444 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:03.574882 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:03.575236 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:03.575241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:03.575245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:03.575441 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:03.576005 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:03.576013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:03.576421 (+ 408us) client_negotiation.cc:769] Sending connection context
0812 02:30:03.576722 (+ 301us) client_negotiation.cc:240] Negotiation successful
0812 02:30:03.577309 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":378,"threads_started":1}
I20260812 02:30:03.580346 20053 heartbeater.cc:347] Connected to a master server at 127.12.8.254:35939
I20260812 02:30:03.580642 20053 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:03.581429 20053 heartbeater.cc:511] Master 127.12.8.254:35939 requested a full tablet report, sending...
I20260812 02:30:03.583783 19826 ts_manager.cc:194] Registered new tserver with Master: 572e2683344e422d87faad60b0bcb6d3 (127.12.8.193:45959)
I20260812 02:30:03.585979 19826 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:52913
WARNING: no policy specified for oryx/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab.
Entry for principal oryx/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab.
I20260812 02:30:03.622855 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:35939
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab
--principal=oryx/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:03.974514 20059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:03.974872 20059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:03.975016 20059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:03.982771 20059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:03.982925 20059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:03.983086 20059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:30:03.993978 20059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab
--principal=oryx/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:35939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:03.995496 20059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:03.997071 20059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:04.012277 20067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:04.012562 20068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:04.012918 20070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:04.013916 20059 server_base.cc:1034] running on GCE node
I20260812 02:30:04.014935 20059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:04.016330 20059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:04.017638 20059 hybrid_clock.cc:648] HybridClock initialized: now 1786501804017560 us; error 45 us; skew 500 ppm
I20260812 02:30:04.018123 20059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:04 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501804, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:04.022528 20059 init.cc:377] Logged in from keytab as oryx/127.12.8.194@KRBTEST.COM (short username oryx)
I20260812 02:30:04.024637 20059 webserver.cc:460] Webserver started at http://127.12.8.194:35237/ using document root <none> and password file <none>
I20260812 02:30:04.025661 20059 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:04.025822 20059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:04.026194 20059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:04.030104 20059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81"
format_stamp: "Formatted at 2026-08-12 02:30:04 on dist-test-slave-vq2q"
server_key: "a7e20682c4bd409b09936bf48af754a5"
server_key_iv: "93fd15828d46f09cb85c766a0b99c8fb"
server_key_version: "encryptionkey@0"
I20260812 02:30:04.031117 20059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81"
format_stamp: "Formatted at 2026-08-12 02:30:04 on dist-test-slave-vq2q"
server_key: "a7e20682c4bd409b09936bf48af754a5"
server_key_iv: "93fd15828d46f09cb85c766a0b99c8fb"
server_key_version: "encryptionkey@0"
I20260812 02:30:04.038127 20059 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260812 02:30:04.042841 20078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:04.044868 20059 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:30:04.045101 20059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81"
format_stamp: "Formatted at 2026-08-12 02:30:04 on dist-test-slave-vq2q"
server_key: "a7e20682c4bd409b09936bf48af754a5"
server_key_iv: "93fd15828d46f09cb85c766a0b99c8fb"
server_key_version: "encryptionkey@0"
I20260812 02:30:04.045444 20059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:04.066431 20059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:04.071910 20059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:04.072279 20059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:04.073555 20059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:04.075284 20059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:04.075378 20059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:04.075493 20059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:04.075543 20059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:04.115767 20059 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:45489
I20260812 02:30:04.115842 20209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:45489 every 8 connection(s)
I20260812 02:30:04.117503 20059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:30:04.122134 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 20059
I20260812 02:30:04.122399 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:30:04.123253 12323 external_mini_cluster.cc:1428] Setting key 8dc82ca8ee976ab123b941dea0dd7e8f
Aug 12 02:30:04 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501804, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:04.143240 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:04.120672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:39405 (local address 127.12.8.254:35939)
0812 02:30:04.121351 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:04.121359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:04.123223 (+ 1864us) server_negotiation.cc:407] Connection header received
0812 02:30:04.125683 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:04.125689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:04.125804 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:04.125969 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:04.128233 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:04.129142 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:04.131179 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:04.131820 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:04.131960 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:04.136519 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:04.136525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:04.136529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:04.136572 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:04.139598 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:04.140501 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:04.140507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:04.140511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:04.140660 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:04.141124 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:04.141127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:04.141130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:04.141492 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:04.141658 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:04.142510 (+ 852us) server_negotiation.cc:299] Negotiation successful
0812 02:30:04.142787 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":289,"threads_started":1}
I20260812 02:30:04.144500 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:04.121461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:39405)
0812 02:30:04.122832 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:04.122930 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:04.125336 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:04.126219 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:04.126232 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:04.127101 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:04.127973 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:04.127998 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:04.129432 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:04.129437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:04.130966 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:04.130986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:04.132157 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:04.132168 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:04.132451 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:04.133303 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:04.133349 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:04.136218 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:04.139781 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:04.139793 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:04.139815 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:04.140365 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:04.140773 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:04.140779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:04.140783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:04.141008 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:04.141631 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:04.141641 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:04.142092 (+ 451us) client_negotiation.cc:769] Sending connection context
0812 02:30:04.142404 (+ 312us) client_negotiation.cc:240] Negotiation successful
0812 02:30:04.143106 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":408,"threads_started":1}
I20260812 02:30:04.146261 20211 heartbeater.cc:347] Connected to a master server at 127.12.8.254:35939
I20260812 02:30:04.146806 20211 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:04.147991 20211 heartbeater.cc:511] Master 127.12.8.254:35939 requested a full tablet report, sending...
I20260812 02:30:04.150336 19826 ts_manager.cc:194] Registered new tserver with Master: 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489)
WARNING: no policy specified for oryx/127.12.8.195@KRBTEST.COM; defaulting to no policy
I20260812 02:30:04.151453 19826 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:39405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab.
Entry for principal oryx/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab.
I20260812 02:30:04.187009 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:35939
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab
--principal=oryx/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:04.543449 20217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:04.543766 20217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:04.543865 20217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:04.551646 20217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:04.551787 20217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:04.551903 20217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:04.562235 20217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab
--principal=oryx/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:35939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:04.563616 20217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:04.564989 20217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:04.580477 20223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:04.580510 20226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:04.580533 20224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:04.581658 20217 server_base.cc:1034] running on GCE node
I20260812 02:30:04.582516 20217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:04.583871 20217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:04.585105 20217 hybrid_clock.cc:648] HybridClock initialized: now 1786501804585052 us; error 46 us; skew 500 ppm
I20260812 02:30:04.585518 20217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:04 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501804, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:04.589144 20217 init.cc:377] Logged in from keytab as oryx/127.12.8.195@KRBTEST.COM (short username oryx)
I20260812 02:30:04.589573 20053 heartbeater.cc:503] Master 127.12.8.254:35939 was elected leader, sending a full tablet report...
I20260812 02:30:04.591343 20217 webserver.cc:460] Webserver started at http://127.12.8.195:39923/ using document root <none> and password file <none>
I20260812 02:30:04.592203 20217 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:04.592356 20217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:04.592700 20217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:04.595980 20217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "78bcbb43058e4f45a71dcb78fc413b2d"
format_stamp: "Formatted at 2026-08-12 02:30:04 on dist-test-slave-vq2q"
server_key: "a5789b787f7584eda3102b9cdd34ebfd"
server_key_iv: "2700c81172cc0d914a806832d0b91011"
server_key_version: "encryptionkey@0"
I20260812 02:30:04.596889 20217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "78bcbb43058e4f45a71dcb78fc413b2d"
format_stamp: "Formatted at 2026-08-12 02:30:04 on dist-test-slave-vq2q"
server_key: "a5789b787f7584eda3102b9cdd34ebfd"
server_key_iv: "2700c81172cc0d914a806832d0b91011"
server_key_version: "encryptionkey@0"
I20260812 02:30:04.603343 20217 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260812 02:30:04.607774 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:04.609524 20217 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:30:04.609716 20217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "78bcbb43058e4f45a71dcb78fc413b2d"
format_stamp: "Formatted at 2026-08-12 02:30:04 on dist-test-slave-vq2q"
server_key: "a5789b787f7584eda3102b9cdd34ebfd"
server_key_iv: "2700c81172cc0d914a806832d0b91011"
server_key_version: "encryptionkey@0"
I20260812 02:30:04.609954 20217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:04.636233 20217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:04.640939 20217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:04.641252 20217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:04.642424 20217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:04.643955 20217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:04.644037 20217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:04.644152 20217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:04.644204 20217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:04.680570 20217 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:37565
I20260812 02:30:04.680735 20362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:37565 every 8 connection(s)
I20260812 02:30:04.682027 20217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:04.688688 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 20217
I20260812 02:30:04.688928 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:30:04.689589 12323 external_mini_cluster.cc:1428] Setting key 8f52b152555faec7893a01b6f71ec1d7
Aug 12 02:30:04 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501804, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:04.707129 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:04.685234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:39501 (local address 127.12.8.254:35939)
0812 02:30:04.685779 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:04.685785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:04.687079 (+ 1294us) server_negotiation.cc:407] Connection header received
0812 02:30:04.689266 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:04.689274 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:04.689418 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:04.689604 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:04.691655 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:04.692912 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:04.694962 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:04.695456 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:04.695612 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:04.699537 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:04.699544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:04.699548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:04.699625 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:04.702756 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:04.703696 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:04.703701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:04.703704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:04.703880 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:04.704443 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:04.704448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:04.704450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:04.704897 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:04.705283 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:04.706259 (+ 976us) server_negotiation.cc:299] Negotiation successful
0812 02:30:04.706613 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
I20260812 02:30:04.707858 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:04.685873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:39501)
0812 02:30:04.686862 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:04.686918 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:04.688794 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:04.689882 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:04.689891 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:04.690579 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:04.691418 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:04.691445 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:04.693218 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:04.693224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:04.694768 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:04.694783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:04.695709 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:04.695714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:04.695921 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:04.696650 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:04.696673 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:04.699203 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:04.702968 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:04.702976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:04.702991 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:04.703501 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:04.704027 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:04.704030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:04.704034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:04.704246 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:04.705103 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:04.705111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:04.705605 (+ 494us) client_negotiation.cc:769] Sending connection context
0812 02:30:04.705967 (+ 362us) client_negotiation.cc:240] Negotiation successful
0812 02:30:04.706551 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":420,"threads_started":1}
I20260812 02:30:04.709542 20363 heartbeater.cc:347] Connected to a master server at 127.12.8.254:35939
I20260812 02:30:04.709836 20363 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:04.710602 20363 heartbeater.cc:511] Master 127.12.8.254:35939 requested a full tablet report, sending...
I20260812 02:30:04.712641 19826 ts_manager.cc:194] Registered new tserver with Master: 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565)
I20260812 02:30:04.713765 19826 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:39501
I20260812 02:30:04.720764 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.8.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.253.keytab.
Entry for principal oryx/127.12.8.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.253.keytab.
I20260812 02:30:04.797876 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.253:37543
--webserver_interface=127.12.8.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.253.keytab
--principal=oryx/127.12.8.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:35939,127.12.8.253:37543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:30:05.155366 20211 heartbeater.cc:503] Master 127.12.8.254:35939 was elected leader, sending a full tablet report...
W20260812 02:30:05.255539 20370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:05.255960 20370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:05.256222 20370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:05.265231 20370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:05.265416 20370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:05.265475 20370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:05.265542 20370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:05.265594 20370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:05.278411 20370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:35939,127.12.8.253:37543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.253.keytab
--principal=oryx/127.12.8.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.253:37543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.8.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
W20260812 02:30:05.278913 20370 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.8.254:35939,127.12.8.253:37543'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 02:30:05.280319 20370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:05.282053 20370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:05.297384 20377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:05.297346 20378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:05.297475 20380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:05.298774 20370 server_base.cc:1034] running on GCE node
I20260812 02:30:05.299932 20370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:05.302042 20370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:05.303318 20370 hybrid_clock.cc:648] HybridClock initialized: now 1786501805303243 us; error 64 us; skew 500 ppm
I20260812 02:30:05.303726 20370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:05 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501805, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:05.307770 20370 init.cc:377] Logged in from keytab as oryx/127.12.8.253@KRBTEST.COM (short username oryx)
I20260812 02:30:05.309917 20370 webserver.cc:460] Webserver started at http://127.12.8.253:46859/ using document root <none> and password file <none>
I20260812 02:30:05.310866 20370 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:05.310994 20370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:05.311403 20370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:05.315457 20370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data/instance:
uuid: "34598cea9ca64d5ea37ded12f60125cf"
format_stamp: "Formatted at 2026-08-12 02:30:05 on dist-test-slave-vq2q"
server_key: "686708045c4c03579922e92a92477a36"
server_key_iv: "5b5cbeb1944476438a56a5553260772e"
server_key_version: "encryptionkey@0"
I20260812 02:30:05.316431 20370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal/instance:
uuid: "34598cea9ca64d5ea37ded12f60125cf"
format_stamp: "Formatted at 2026-08-12 02:30:05 on dist-test-slave-vq2q"
server_key: "686708045c4c03579922e92a92477a36"
server_key_iv: "5b5cbeb1944476438a56a5553260772e"
server_key_version: "encryptionkey@0"
I20260812 02:30:05.323721 20370 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260812 02:30:05.328742 20390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:05.331072 20370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:30:05.331377 20370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
uuid: "34598cea9ca64d5ea37ded12f60125cf"
format_stamp: "Formatted at 2026-08-12 02:30:05 on dist-test-slave-vq2q"
server_key: "686708045c4c03579922e92a92477a36"
server_key_iv: "5b5cbeb1944476438a56a5553260772e"
server_key_version: "encryptionkey@0"
I20260812 02:30:05.331719 20370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:05.351202 20370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:05.356559 20370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:05.356880 20370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:05.378624 20370 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.253:37543
I20260812 02:30:05.378628 20450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.253:37543 every 8 connection(s)
I20260812 02:30:05.380574 20370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
I20260812 02:30:05.385667 20451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:05.386763 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 20370
I20260812 02:30:05.398672 20451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:30:05 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.253@KRBTEST.COM
Aug 12 02:30:05 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501805, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.253@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:05.423604 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.389738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.12.8.253:37543)
0812 02:30:05.390872 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:05.390898 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:05.390968 (+ 70us) server_negotiation.cc:407] Connection header received
0812 02:30:05.394278 (+ 3310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:05.394330 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:05.395230 (+ 900us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:05.395928 (+ 698us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:05.397857 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.400126 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.401803 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.402891 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.403131 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.408955 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:05.408966 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:05.408981 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:05.409073 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:05.417397 (+ 8324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.418700 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.418710 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.418721 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.418955 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.419563 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.419567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.419569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.420501 (+ 932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:05.420802 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:05.421554 (+ 752us) server_negotiation.cc:299] Negotiation successful
0812 02:30:05.422187 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":473,"threads_started":1}
I20260812 02:30:05.426451 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.402881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:60136)
0812 02:30:05.404160 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:05.404221 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:05.404434 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:05.404926 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:05.404935 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:05.405984 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:05.406399 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.406412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.407714 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.407720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.409364 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.409378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.410155 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.410162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.410381 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.415339 (+ 4958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:05.415374 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:05.418801 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:05.423074 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.423088 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.423097 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.423616 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.424089 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.424095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.424099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.424272 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.425012 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:05.425020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:05.425327 (+ 307us) client_negotiation.cc:769] Sending connection context
0812 02:30:05.425583 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 02:30:05.426050 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":711,"threads_started":1}
I20260812 02:30:05.427461 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.402887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.12.8.254:35939)
0812 02:30:05.404156 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:05.404163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:05.404333 (+ 170us) server_negotiation.cc:407] Connection header received
0812 02:30:05.404601 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:05.404605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:05.404700 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:05.404818 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:05.406595 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.407533 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.409518 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.409961 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.410111 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.419082 (+ 8971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:05.419091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:05.419096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:05.419169 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:05.422896 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.423775 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.423782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.423786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.423966 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.424399 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.424403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.424405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.424845 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:05.425015 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:05.426841 (+ 1826us) server_negotiation.cc:299] Negotiation successful
0812 02:30:05.427110 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":1166,"threads_started":1}
I20260812 02:30:05.429387 20451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:30:05 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501805, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.253@KRBTEST.COM for oryx/127.12.8.253@KRBTEST.COM
I20260812 02:30:05.443602 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.430152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.0.0.1:34940)
0812 02:30:05.430346 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:05.430364 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:05.430462 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:05.430709 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:05.430713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:05.431108 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:05.431308 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.431318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.432503 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.432507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.433622 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.433638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.434440 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.434446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.434616 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.435226 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:05.435245 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:05.437781 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:05.441105 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.441110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.441113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.441592 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.441948 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.441951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.441954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.442084 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.442676 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:05.442680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:05.442837 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 02:30:05.442967 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 02:30:05.443303 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 02:30:05.443794 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.430334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.12.8.253:37543)
0812 02:30:05.430473 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:05.430478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:05.430494 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:30:05.430537 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:05.430540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:05.430604 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:05.430686 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:05.431490 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.432351 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.433776 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.434265 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.434404 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.438076 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:05.438148 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:05.438153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:05.438198 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:05.440944 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.441709 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.441714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.441716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.441856 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.442195 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.442198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.442200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.442547 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:05.442674 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:05.443092 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 02:30:05.443469 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":44}
I20260812 02:30:05.449713 20451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Bootstrap starting.
I20260812 02:30:05.453977 20451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:05.455382 20451 log.cc:824] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:05.458956 20451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: No bootstrap required, opened a new log
I20260812 02:30:05.467243 20451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } }
I20260812 02:30:05.467756 20451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:05.467865 20451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34598cea9ca64d5ea37ded12f60125cf, State: Initialized, Role: FOLLOWER
I20260812 02:30:05.468600 20451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } }
I20260812 02:30:05.469739 20459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } } }
I20260812 02:30:05.470103 20459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:30:05.471048 20451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:30:05.482190 20471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:30:05.482403 20471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:30:05.488332 20370 master_runner.cc:428] Detected that this master 34598cea9ca64d5ea37ded12f60125cf is joining an existing cluster
I20260812 02:30:05.488487 20370 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.8.253:37543
I20260812 02:30:05.517315 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.494928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:60138)
0812 02:30:05.495712 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:05.495738 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:05.496008 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:05.496907 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:05.496913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:05.497862 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:05.498364 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.498380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.500064 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.500074 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.501952 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.501971 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.504484 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.504495 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.504754 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.505829 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:05.505869 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:05.507323 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:05.512318 (+ 4995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.512325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.512329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.513112 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.514323 (+ 1211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.514330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.514334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.514552 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.515640 (+ 1088us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:05.515652 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:05.515931 (+ 279us) client_negotiation.cc:769] Sending connection context
0812 02:30:05.516262 (+ 331us) client_negotiation.cc:240] Negotiation successful
0812 02:30:05.516696 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":415,"threads_started":1}
I20260812 02:30:05.518554 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.494659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.12.8.254:35939)
0812 02:30:05.494903 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:05.494910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:05.495844 (+ 934us) server_negotiation.cc:407] Connection header received
0812 02:30:05.496378 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:05.496385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:05.496514 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:05.496607 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:05.498582 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.499809 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.502166 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.503702 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.503865 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.507661 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:05.507672 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:05.507679 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:05.507738 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:05.512084 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.513805 (+ 1721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.513815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.513819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.514068 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.514727 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.514732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.514736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.515360 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:05.515561 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:05.517779 (+ 2218us) server_negotiation.cc:299] Negotiation successful
0812 02:30:05.518154 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:30:05.518647 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.496050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.12.8.253:37543)
0812 02:30:05.496244 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:05.496255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:05.501314 (+ 5059us) server_negotiation.cc:407] Connection header received
0812 02:30:05.501540 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:05.501545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:05.501613 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:05.501686 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:05.503921 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.505274 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.507196 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.507985 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.508094 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.510991 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:05.510997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:05.511001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:05.511035 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:05.514852 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.515690 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.515709 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.515712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.516052 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.516468 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.516472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.516475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.516832 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:05.517008 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:05.517795 (+ 787us) server_negotiation.cc:299] Negotiation successful
0812 02:30:05.518072 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":45440}
I20260812 02:30:05.520188 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.496235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.0.0.1:34948)
0812 02:30:05.501200 (+ 4965us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:05.501222 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:05.501416 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:05.501885 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:05.501890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:05.502694 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:05.503723 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.503740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.505516 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.505520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.507013 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.507035 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.508413 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.508419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.508579 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.509281 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:05.509303 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:05.510780 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:05.515041 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.515045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.515048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.515560 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.516193 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.516197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.516200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.516359 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.517292 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:05.517301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:05.517476 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 02:30:05.518952 (+ 1476us) client_negotiation.cc:240] Negotiation successful
0812 02:30:05.519342 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4786,"thread_start_us":15645,"threads_started":1}
WARNING: no policy specified for oryx/127.12.8.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.252@KRBTEST.COM" created.
Aug 12 02:30:05 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.254@KRBTEST.COM for oryx/127.12.8.253@KRBTEST.COM
I20260812 02:30:05.551093 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.532811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.0.0.1:34962)
0812 02:30:05.533687 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:05.533754 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:05.534062 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:05.534979 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:05.534988 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:05.536093 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:05.536457 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.536472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.537935 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.537941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.539109 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.539124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.539955 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.539962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.540221 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.540958 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:05.540979 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:05.543659 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:05.547164 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.547181 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.547191 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.547886 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.548417 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.548422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.548426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.548641 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.549648 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:05.549658 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:05.549912 (+ 254us) client_negotiation.cc:769] Sending connection context
0812 02:30:05.550142 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 02:30:05.550657 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":821,"threads_started":1}
I20260812 02:30:05.551080 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.533427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.12.8.253:37543)
0812 02:30:05.533644 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:05.533650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:05.533883 (+ 233us) server_negotiation.cc:407] Connection header received
0812 02:30:05.534338 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:05.534346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:05.534512 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:05.534618 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:05.536699 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.537774 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.539263 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.539776 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.539896 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.543900 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:05.543906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:05.543913 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:05.543954 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:05.546972 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.548083 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.548089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.548093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.548285 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.548794 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.548800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.548803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.549391 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:05.549565 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:05.550270 (+ 705us) server_negotiation.cc:299] Negotiation successful
0812 02:30:05.550586 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 02:30:05.554320 19826 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.8.253:37543: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf"
member_type: NON_VOTER
last_known_addr {
host: "127.12.8.253"
port: 37543
}
attrs {
promote: true
}
}
dest_uuid: "c787dfba77e6402dadf24679cfedd84b"
cas_config_opid_index: -1
I20260812 02:30:05.556496 19826 consensus_queue.cc:237] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } }
I20260812 02:30:05.572247 20485 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 34598cea9ca64d5ea37ded12f60125cf (127.12.8.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:05.573982 20420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:05.577328 20485 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.8.253:37543
I20260812 02:30:05.578378 20484 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:05.578691 20370 master.cc:537] Master@127.12.8.253:37543 shutting down...
I20260812 02:30:05.578848 20420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Refusing update from remote peer c787dfba77e6402dadf24679cfedd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.252.keytab.
Entry for principal oryx/127.12.8.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.252.keytab.
I20260812 02:30:05.579831 20484 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
W20260812 02:30:05.581111 12323 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.12.8.252: real 0.100s user 0.005s sys 0.027s
I20260812 02:30:05.581470 20485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:05.582587 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.252:35599
--webserver_interface=127.12.8.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.252.keytab
--principal=oryx/127.12.8.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:35939,127.12.8.253:37543,127.12.8.252:35599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:05.585444 19811 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 02:30:05.594664 19811 consensus_peers.cc:487] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b -> Peer 34598cea9ca64d5ea37ded12f60125cf (127.12.8.253:37543): Couldn't send request to peer 34598cea9ca64d5ea37ded12f60125cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 02:30:05.606566 20370 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:05.607362 20370 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:05.607692 20370 tablet_replica.cc:330] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: stopping tablet replica
I20260812 02:30:05.617426 20370 master.cc:552] Master@127.12.8.253:37543 shutdown complete.
I20260812 02:30:05.617669 20370 master_runner.cc:305] Clearing existing system tablet
I20260812 02:30:05.618283 20370 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:05.618399 20370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:05.621323 20370 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:30:05.623369 20490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:05.623788 20370 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:05.624089 20370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:30:05.624296 20370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
uuid: "34598cea9ca64d5ea37ded12f60125cf"
format_stamp: "Formatted at 2026-08-12 02:30:05 on dist-test-slave-vq2q"
server_key: "686708045c4c03579922e92a92477a36"
server_key_iv: "5b5cbeb1944476438a56a5553260772e"
server_key_version: "encryptionkey@0"
I20260812 02:30:05.633528 20370 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:05.640686 20370 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 02:30:05.641090 20370 log.cc:1196] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 02:30:05.641836 20370 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Deleting consensus metadata
I20260812 02:30:05.642261 20370 master_runner.cc:315] Copying system tablet from 127.12.8.254:35939
I20260812 02:30:05.663802 20370 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: tablet copy: Beginning tablet copy session from remote peer at address 127.12.8.254:35939
I20260812 02:30:05.686093 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.665245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:60140)
0812 02:30:05.666045 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:05.666075 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:05.666373 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:05.667069 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:05.667075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:05.667758 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:05.668188 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.668206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.669928 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.669935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.671702 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.671721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.673335 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.673341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.673567 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.674505 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:05.674536 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:05.677008 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:05.682364 (+ 5356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.682370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.682376 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.683015 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.683651 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:05.683656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:05.683660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:05.683852 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:05.684762 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:05.684767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:05.684941 (+ 174us) client_negotiation.cc:769] Sending connection context
0812 02:30:05.685097 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 02:30:05.685558 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":471,"threads_started":1}
I20260812 02:30:05.686143 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.665346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.12.8.254:35939)
0812 02:30:05.665584 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:05.665590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:05.666206 (+ 616us) server_negotiation.cc:407] Connection header received
0812 02:30:05.666562 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:05.666570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:05.666693 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:05.666783 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:05.668413 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.669684 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.671903 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.672728 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.672849 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.677607 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:05.677618 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:05.677624 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:05.677711 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:05.682093 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.683219 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.683225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.683228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.683427 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:05.683997 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:05.684003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:05.684006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:05.684471 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:05.684645 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:05.685277 (+ 632us) server_negotiation.cc:299] Negotiation successful
0812 02:30:05.685548 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 02:30:05.687510 19852 tablet_copy_service.cc:140] P c787dfba77e6402dadf24679cfedd84b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 34598cea9ca64d5ea37ded12f60125cf ({username='oryx', principal='oryx/127.12.8.253@KRBTEST.COM'} at 127.0.0.1:60140)
I20260812 02:30:05.687963 19852 tablet_copy_service.cc:161] P c787dfba77e6402dadf24679cfedd84b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 34598cea9ca64d5ea37ded12f60125cf at {username='oryx', principal='oryx/127.12.8.253@KRBTEST.COM'} at 127.0.0.1:60140: session id = 34598cea9ca64d5ea37ded12f60125cf-00000000000000000000000000000000
I20260812 02:30:05.692490 19852 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 02:30:05.695674 20370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 02:30:05.702275 20370 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: tablet copy: Starting download of 0 data blocks...
I20260812 02:30:05.702737 20370 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: tablet copy: Starting download of 1 WAL segments...
I20260812 02:30:05.706842 20370 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 02:30:05.711210 19852 tablet_copy_service.cc:342] P c787dfba77e6402dadf24679cfedd84b: Request end of tablet copy session 34598cea9ca64d5ea37ded12f60125cf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.8.253@KRBTEST.COM'} at 127.0.0.1:60140
I20260812 02:30:05.711614 19852 tablet_copy_service.cc:433] P c787dfba77e6402dadf24679cfedd84b: ending tablet copy session 34598cea9ca64d5ea37ded12f60125cf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 34598cea9ca64d5ea37ded12f60125cf
I20260812 02:30:05.716922 20363 heartbeater.cc:503] Master 127.12.8.254:35939 was elected leader, sending a full tablet report...
I20260812 02:30:05.727155 20370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:05.730225 20498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:05.741016 20501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:05.741413 20370 server_base.cc:1034] running on GCE node
W20260812 02:30:05.741628 20499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:05.742259 20370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:05.743002 20370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:05.744270 20370 hybrid_clock.cc:648] HybridClock initialized: now 1786501805744198 us; error 44 us; skew 500 ppm
I20260812 02:30:05.744555 20370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:05 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501805, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:05.746781 20370 init.cc:377] Logged in from keytab as oryx/127.12.8.253@KRBTEST.COM (short username oryx)
I20260812 02:30:05.748313 20370 webserver.cc:460] Webserver started at http://127.12.8.253:35495/ using document root <none> and password file <none>
I20260812 02:30:05.748694 20370 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:05.748771 20370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:05.751935 20370 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:30:05.753990 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:05.754463 20370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 02:30:05.754621 20370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
uuid: "34598cea9ca64d5ea37ded12f60125cf"
format_stamp: "Formatted at 2026-08-12 02:30:05 on dist-test-slave-vq2q"
server_key: "686708045c4c03579922e92a92477a36"
server_key_iv: "5b5cbeb1944476438a56a5553260772e"
server_key_version: "encryptionkey@0"
I20260812 02:30:05.754796 20370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:05.767546 20370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:05.771255 20370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:05.827072 20370 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.253:37543
I20260812 02:30:05.827121 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.253:37543 every 8 connection(s)
I20260812 02:30:05.828528 20370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
I20260812 02:30:05.835860 20567 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:30:05.840344 20567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Bootstrap starting.
I20260812 02:30:05.865446 20567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:30:05.866199 20567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Bootstrap complete.
I20260812 02:30:05.867449 20567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } }
I20260812 02:30:05.867743 20567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34598cea9ca64d5ea37ded12f60125cf, State: Initialized, Role: LEARNER
I20260812 02:30:05.868072 20567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } }
I20260812 02:30:05.875152 20570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:05.876412 20567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:05.877713 20570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: This master's current role is: LEARNER
I20260812 02:30:05.901288 20584 catalog_manager.cc:1259] Loaded cluster ID: f2332070953147fab8eb91c78e45377b
I20260812 02:30:05.901398 20584 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: loading cluster ID for follower catalog manager: success
I20260812 02:30:05.904098 20584 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: acquiring CA information for follower catalog manager: success
I20260812 02:30:05.906268 20584 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:30:05.983487 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.976868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.0.0.1:34970)
0812 02:30:05.977219 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:05.977240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:05.977384 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:05.978344 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:05.978349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:05.978389 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:05.978592 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.978603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.979997 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.980002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.981346 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:05.981360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.982621 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.982627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:05.982854 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.982917 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 02:30:05.983006 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 02:30:05.983152 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260812 02:30:05.983548 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:05.977343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.12.8.253:37543)
0812 02:30:05.977981 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:05.977986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:05.978004 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:30:05.978076 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:05.978080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:05.978154 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:05.978264 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:05.978755 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.979854 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.981509 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:05.982448 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:05.982586 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:05.982865 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:05.983048 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 02:30:05.983184 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":327,"threads_started":1}
I20260812 02:30:05.984962 20538 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260812 02:30:05.985606 20570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c787dfba77e6402dadf24679cfedd84b. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:05.985922 20570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: This master's current role is: LEARNER
W20260812 02:30:06.214385 20488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:06.214839 20488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:06.215101 20488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:06.226377 20488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:06.226634 20488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:06.226787 20488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:06.226958 20488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:06.227082 20488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:06.242406 20488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:35939,127.12.8.253:37543,127.12.8.252:35599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.252.keytab
--principal=oryx/127.12.8.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.252:35599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.8.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:06.244336 20488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:06.246152 20488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:06.264628 20590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:06.265498 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:06.265626 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:06.267601 20488 server_base.cc:1034] running on GCE node
I20260812 02:30:06.268656 20488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:06.270596 20488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:06.271860 20488 hybrid_clock.cc:648] HybridClock initialized: now 1786501806271745 us; error 53 us; skew 500 ppm
I20260812 02:30:06.272348 20488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:06 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501806, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:06.277514 20488 init.cc:377] Logged in from keytab as oryx/127.12.8.252@KRBTEST.COM (short username oryx)
I20260812 02:30:06.279770 20488 webserver.cc:460] Webserver started at http://127.12.8.252:43931/ using document root <none> and password file <none>
I20260812 02:30:06.280714 20488 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:06.280972 20488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:06.281458 20488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:06.285872 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data/instance:
uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
format_stamp: "Formatted at 2026-08-12 02:30:06 on dist-test-slave-vq2q"
server_key: "65550bd8ec958ec23552ae125ebe9f8e"
server_key_iv: "74655b0ec3eb349ab9ef73271f0cdab7"
server_key_version: "encryptionkey@0"
I20260812 02:30:06.286895 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal/instance:
uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
format_stamp: "Formatted at 2026-08-12 02:30:06 on dist-test-slave-vq2q"
server_key: "65550bd8ec958ec23552ae125ebe9f8e"
server_key_iv: "74655b0ec3eb349ab9ef73271f0cdab7"
server_key_version: "encryptionkey@0"
I20260812 02:30:06.294438 20488 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260812 02:30:06.299841 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:06.301677 20488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:30:06.301863 20488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
format_stamp: "Formatted at 2026-08-12 02:30:06 on dist-test-slave-vq2q"
server_key: "65550bd8ec958ec23552ae125ebe9f8e"
server_key_iv: "74655b0ec3eb349ab9ef73271f0cdab7"
server_key_version: "encryptionkey@0"
I20260812 02:30:06.302184 20488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:06.319710 20488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:06.325703 20488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:06.326181 20488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:06.372200 20488 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.252:35599
I20260812 02:30:06.372354 20661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.252:35599 every 8 connection(s)
I20260812 02:30:06.374197 20488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
I20260812 02:30:06.379923 20662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:06.382217 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 20488
I20260812 02:30:06.391611 20662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:30:06 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.252@KRBTEST.COM
Aug 12 02:30:06 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501806, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.252@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:06.410844 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.384999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.12.8.252:35599)
0812 02:30:06.386453 (+ 1454us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.386466 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.386511 (+ 45us) server_negotiation.cc:407] Connection header received
0812 02:30:06.388649 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.388674 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.389133 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.389587 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.391815 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.393378 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.395235 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.396379 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.396546 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.401950 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.401960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.401973 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.402009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.405893 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.406668 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.406677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.406687 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.406935 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.407388 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.407393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.407396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.408037 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.408328 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.408804 (+ 476us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.409303 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"spinlock_wait_cycles":13568,"thread_start_us":870,"threads_started":1}
I20260812 02:30:06.413838 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.394682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:60142)
0812 02:30:06.396221 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:06.396261 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:06.396453 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:06.396922 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:06.396929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:06.397960 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:06.398366 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.398390 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.399409 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.399416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.400916 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.400933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.401733 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.401741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.401969 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.402637 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:06.402695 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:06.405796 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:06.409839 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.409849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.409855 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.410560 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.411122 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.411127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.411131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.411337 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.412488 (+ 1151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:06.412495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:06.412714 (+ 219us) client_negotiation.cc:769] Sending connection context
0812 02:30:06.412991 (+ 277us) client_negotiation.cc:240] Negotiation successful
0812 02:30:06.413356 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"thread_start_us":1075,"threads_started":1}
I20260812 02:30:06.413889 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.394674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.12.8.254:35939)
0812 02:30:06.395326 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.395333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.396372 (+ 1039us) server_negotiation.cc:407] Connection header received
0812 02:30:06.396606 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.396612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.396705 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.396845 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.398546 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.399257 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.401062 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.401508 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.401602 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.406031 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.406038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.406043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.406086 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.409716 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.410706 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.410711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.410713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.410964 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.411853 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.411858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.411862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.412350 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.412506 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.413077 (+ 571us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.413416 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":424,"threads_started":1}
I20260812 02:30:06.415452 20662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:30:06 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501806, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.252@KRBTEST.COM for oryx/127.12.8.253@KRBTEST.COM
I20260812 02:30:06.431780 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.416083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.0.0.1:34974)
0812 02:30:06.416261 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:06.416282 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:06.416406 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:06.416712 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:06.416717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:06.417150 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:06.417362 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.417377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.418455 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.418459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.419601 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.419614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.420463 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.420474 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.420625 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.421209 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:06.421253 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:06.424508 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:06.428761 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.428767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.428772 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.429456 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.429839 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.429842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.429844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.430068 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.430712 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:06.430716 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:06.430900 (+ 184us) client_negotiation.cc:769] Sending connection context
0812 02:30:06.431030 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 02:30:06.431358 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 02:30:06.431895 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.416218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.12.8.253:37543)
0812 02:30:06.416395 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.416401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.416421 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:30:06.416519 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.416525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.416597 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.416679 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.417509 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.418274 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.419763 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.420304 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.420411 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.424729 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.424735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.424739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.424778 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.428614 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.429557 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.429562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.429565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.429756 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.430210 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.430214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.430216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.430578 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.430706 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.431144 (+ 438us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.431449 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:30:06.433044 20662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:30:06 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501806, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.252@KRBTEST.COM for oryx/127.12.8.252@KRBTEST.COM
I20260812 02:30:06.449961 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.434508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.12.8.252:35599)
0812 02:30:06.434707 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.434711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.434727 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:30:06.434765 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.434769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.434822 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.434899 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.435750 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.436473 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.438113 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.438563 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.438662 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.443067 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.443072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.443075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.443104 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.446719 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.447502 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.447507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.447510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.447701 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.448201 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.448205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.448207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.448597 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.448728 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.449210 (+ 482us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.449532 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":78}
I20260812 02:30:06.450001 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.434248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:58360)
0812 02:30:06.434492 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:06.434508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:06.434593 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:06.434933 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:06.434937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:06.435377 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:06.435635 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.435645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.436593 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.436597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.437980 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.437992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.438692 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.438696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.438981 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.439741 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:06.439760 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:06.442811 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:06.446868 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.446874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.446879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.447396 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.447809 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.447814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.447817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.448091 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.448707 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:06.448712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:06.448923 (+ 211us) client_negotiation.cc:769] Sending connection context
0812 02:30:06.449048 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 02:30:06.449357 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 02:30:06.457059 20662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Bootstrap starting.
I20260812 02:30:06.461783 20662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:06.463789 20662 log.cc:824] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:06.467775 20662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: No bootstrap required, opened a new log
I20260812 02:30:06.480309 20662 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:30:06.480881 20662 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:06.481103 20662 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8f0ac0a76142949e22cd16f7e51fce, State: Initialized, Role: FOLLOWER
I20260812 02:30:06.481828 20662 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:30:06.483620 20672 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:30:06.484342 20672 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:30:06.484766 20662 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:30:06.500430 20687 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:30:06.500679 20687 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:30:06.505203 20488 master_runner.cc:428] Detected that this master ed8f0ac0a76142949e22cd16f7e51fce is joining an existing cluster
I20260812 02:30:06.505438 20488 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.8.252:35599
I20260812 02:30:06.518728 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 19902
I20260812 02:30:06.538985 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.513162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.12.8.254:35939)
0812 02:30:06.513457 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.513463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.515613 (+ 2150us) server_negotiation.cc:407] Connection header received
0812 02:30:06.515980 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.515988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.516091 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.516176 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.518435 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.519536 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.522296 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.522996 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.523103 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.526595 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.526604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.526609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.526655 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.530857 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.533336 (+ 2479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.533342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.533346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.533934 (+ 588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.534809 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.534814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.534817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.535319 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.535502 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.538161 (+ 2659us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.538456 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260812 02:30:06.539853 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.513538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:60150)
0812 02:30:06.515476 (+ 1938us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:06.515499 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:06.515795 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:06.516461 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:06.516742 (+ 281us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:06.517785 (+ 1043us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:06.518243 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.518258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.520103 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.520109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.522103 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.522124 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.523505 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.523517 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.523714 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.524883 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:06.524906 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:06.526313 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:06.531056 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.531061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.531065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.533036 (+ 1971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.534103 (+ 1067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.534108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.534112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.534421 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.535803 (+ 1382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:06.535807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:06.536226 (+ 419us) client_negotiation.cc:769] Sending connection context
0812 02:30:06.537989 (+ 1763us) client_negotiation.cc:240] Negotiation successful
0812 02:30:06.538588 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1738,"thread_start_us":1241,"threads_started":1}
I20260812 02:30:06.547871 20672 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:06.548494 20672 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:30:06.550356 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.513210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.12.8.252:35599)
0812 02:30:06.513429 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.513437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.523546 (+ 10109us) server_negotiation.cc:407] Connection header received
0812 02:30:06.523949 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.523954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.524020 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.524085 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.526045 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.527367 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.530179 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.531046 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.531154 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.534977 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.534984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.534990 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.535025 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.541336 (+ 6311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.542329 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.542334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.542356 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.542631 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.543101 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.543110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.543114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.546384 (+ 3270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.546556 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.548033 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.548302 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:30:06.551682 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.514202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:58366)
0812 02:30:06.523430 (+ 9228us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:06.523451 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:06.523821 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:06.524251 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:06.524255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:06.525223 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:06.525882 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.525901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.527608 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.527614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.529961 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.530020 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.531671 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.531679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.532009 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.532952 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:06.533000 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:06.534769 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:06.541491 (+ 6722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.541501 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.541505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.542218 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.542774 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.542782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.542785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.542980 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.546611 (+ 3631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:06.546616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:06.546801 (+ 185us) client_negotiation.cc:769] Sending connection context
0812 02:30:06.547015 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 02:30:06.547467 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9078,"thread_start_us":16902,"threads_started":1}
I20260812 02:30:06.552206 20672 leader_election.cc:288] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939), 34598cea9ca64d5ea37ded12f60125cf (127.12.8.253:37543)
I20260812 02:30:06.553923 19839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c787dfba77e6402dadf24679cfedd84b" is_pre_election: true
I20260812 02:30:06.554934 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34598cea9ca64d5ea37ded12f60125cf" is_pre_election: true
Aug 12 02:30:06 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.193@KRBTEST.COM
I20260812 02:30:06.568513 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.520325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.12.8.193:45959)
0812 02:30:06.537265 (+ 16940us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.537276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.537311 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:30:06.537400 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.537410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.537688 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.537847 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.543225 (+ 5378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.544801 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.547498 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.548955 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.549098 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.557666 (+ 8568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.557692 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.557713 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.557793 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.564144 (+ 6351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.565208 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.565220 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.565224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.565478 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.566026 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.566032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.566036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.566647 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.567086 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.567436 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.567812 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16692,"thread_start_us":16428,"threads_started":1}
I20260812 02:30:06.570411 20045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:06.588984 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:06.590050 20609 leader_election.cc:302] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed8f0ac0a76142949e22cd16f7e51fce; no voters: 34598cea9ca64d5ea37ded12f60125cf, c787dfba77e6402dadf24679cfedd84b
I20260812 02:30:06.590620 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
W20260812 02:30:06.589612 20611 tablet.cc:2369] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:30:06.591691 20672 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 12 02:30:06 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.254@KRBTEST.COM for oryx/127.12.8.252@KRBTEST.COM
I20260812 02:30:06.607481 20702 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: attempting to promote NON_VOTER 34598cea9ca64d5ea37ded12f60125cf to VOTER
I20260812 02:30:06.609241 20702 consensus_queue.cc:237] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } }
I20260812 02:30:06.617527 20538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 LEARNER]: Refusing update from remote peer c787dfba77e6402dadf24679cfedd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 02:30:06.618501 20699 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 02:30:06.619347 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.516264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.0.0.1:34976)
0812 02:30:06.534585 (+ 18321us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:06.534608 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:06.535641 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:06.537051 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:06.537056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:06.540313 (+ 3257us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:06.540997 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.541014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.542875 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.542880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.552854 (+ 9974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.552873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.554810 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.554820 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.555228 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.561446 (+ 6218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:06.561480 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:06.593821 (+ 32341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:06.607107 (+ 13286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.607114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.607118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.607866 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.616424 (+ 8558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.616429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.616432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.616638 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.617569 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:06.617574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:06.617889 (+ 315us) client_negotiation.cc:769] Sending connection context
0812 02:30:06.618035 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 02:30:06.618515 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14699,"thread_start_us":7297,"threads_started":1}
I20260812 02:30:06.621037 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.515918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.12.8.253:37543)
0812 02:30:06.516119 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.516125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.534704 (+ 18579us) server_negotiation.cc:407] Connection header received
0812 02:30:06.536549 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.536556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.536654 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.536731 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.541234 (+ 4503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.542659 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.553073 (+ 10414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.554132 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.554240 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.594173 (+ 39933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.594183 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.594187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.594256 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.606132 (+ 11876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.608191 (+ 2059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.608197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.608200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.608445 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.616804 (+ 8359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.616809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.616812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.617322 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.619826 (+ 2504us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.620021 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.620414 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:30:06.624567 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.561471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.12.8.252:35599)
0812 02:30:06.561630 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.561634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.561649 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:30:06.561719 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.561722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.561834 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.561934 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.589577 (+ 27643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.593743 (+ 4166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.596099 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.597477 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.597625 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.609027 (+ 11402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.609039 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.609044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.609092 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.614707 (+ 5615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.616016 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.616021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.616025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.616230 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.621888 (+ 5658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.621893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.621897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.622425 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.622595 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.623604 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.623934 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:30:06.626143 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.557825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:58370)
0812 02:30:06.558906 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:06.558951 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:06.559096 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:06.561991 (+ 2895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:06.561996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:06.562799 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:06.563395 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.563407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.593989 (+ 30582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.593996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.595866 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.595889 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.597653 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.597661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.598661 (+ 1000us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.599570 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:06.599596 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:06.608704 (+ 9108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:06.614951 (+ 6247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.614956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.614960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.615847 (+ 887us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.621435 (+ 5588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.621441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.621444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.621730 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.622673 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:06.622679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:06.622927 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 02:30:06.623088 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 02:30:06.625473 (+ 2385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":634,"threads_started":1}
I20260812 02:30:06.628507 19826 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.8.252:35599: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
member_type: NON_VOTER
last_known_addr {
host: "127.12.8.252"
port: 35599
}
attrs {
promote: true
}
}
dest_uuid: "c787dfba77e6402dadf24679cfedd84b"
cas_config_opid_index: 5
W20260812 02:30:06.629473 19826 master_service.cc:301] AddMaster 127.12.8.252:35599 failed with retriable error: Illegal state: Failed initiating master Raft ChangeConfig request, error: unknown: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } }
I20260812 02:30:06.631223 20699 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 34598cea9ca64d5ea37ded12f60125cf (127.12.8.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } }
W20260812 02:30:06.631819 20692 master_proxy_rpc.cc:203] Re-attempting AddMaster request to leader Master (127.12.8.254:35939)
I20260812 02:30:06.637092 20703 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } }
I20260812 02:30:06.637837 20703 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:06.638314 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } }
I20260812 02:30:06.638660 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:06.640709 20536 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 34598cea9ca64d5ea37ded12f60125cf (127.12.8.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } }
I20260812 02:30:06.647583 19826 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.8.252:35599: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
member_type: NON_VOTER
last_known_addr {
host: "127.12.8.252"
port: 35599
}
attrs {
promote: true
}
}
dest_uuid: "c787dfba77e6402dadf24679cfedd84b"
cas_config_opid_index: 6
I20260812 02:30:06.648648 19826 consensus_queue.cc:237] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } }
I20260812 02:30:06.652689 20634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:06.655833 20538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Refusing update from remote peer c787dfba77e6402dadf24679cfedd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 02:30:06.655604 20707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } }
I20260812 02:30:06.656152 20707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:30:06.657424 20702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 02:30:06.658785 20634 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 FOLLOWER]: Refusing update from remote peer c787dfba77e6402dadf24679cfedd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 02:30:06.659924 20702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:06.661724 20703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:06.666926 20703 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.8.252:35599
I20260812 02:30:06.667749 20488 master.cc:537] Master@127.12.8.252:35599 shutting down...
I20260812 02:30:06.667861 20538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:06.669446 20703 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:06.669838 20703 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:06.669539 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:06.670099 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:06.673547 20707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:06.674141 20707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:30:06.681735 20634 raft_consensus.cc:858] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } }
W20260812 02:30:06.684304 20634 raft_consensus.cc:1565] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } }
I20260812 02:30:06.699396 20634 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 34598cea9ca64d5ea37ded12f60125cf (127.12.8.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:06.705664 20710 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:06.706485 20710 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 02:30:06.706748 20672 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c787dfba77e6402dadf24679cfedd84b. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:06.706589 20702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:06.706979 20672 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 02:30:06.707062 20702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:06.707309 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: NON_VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: true } } }
I20260812 02:30:06.707762 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260812 02:30:06.710214 19809 consensus_peers.cc:487] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b -> Peer ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252:35599): Couldn't send request to peer ed8f0ac0a76142949e22cd16f7e51fce. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 02:30:06.710606 20488 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260812 02:30:06.711136 20488 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260812 02:30:06.711324 20488 tablet_replica.cc:330] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: stopping tablet replica
I20260812 02:30:06.719930 20488 master.cc:552] Master@127.12.8.252:35599 shutdown complete.
I20260812 02:30:06.720111 20488 master_runner.cc:305] Clearing existing system tablet
I20260812 02:30:06.720604 20488 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:06.720715 20488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:06.727231 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 19902
I20260812 02:30:06.729140 20488 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 02:30:06.737699 20716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:06.738233 20488 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:06.738513 20488 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.008s sys 0.000s
I20260812 02:30:06.738651 20488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
format_stamp: "Formatted at 2026-08-12 02:30:06 on dist-test-slave-vq2q"
server_key: "65550bd8ec958ec23552ae125ebe9f8e"
server_key_iv: "74655b0ec3eb349ab9ef73271f0cdab7"
server_key_version: "encryptionkey@0"
I20260812 02:30:06.743359 20488 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:06.750385 20488 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 02:30:06.750602 20488 log.cc:1196] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 02:30:06.751190 20488 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Deleting consensus metadata
I20260812 02:30:06.751430 20488 master_runner.cc:315] Copying system tablet from 127.12.8.254:35939
I20260812 02:30:06.756592 20488 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: tablet copy: Beginning tablet copy session from remote peer at address 127.12.8.254:35939
I20260812 02:30:06.760955 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:45959
--local_ip_for_outbound_sockets=127.12.8.193
--tserver_master_addrs=127.12.8.254:35939
--webserver_port=38951
--webserver_interface=127.12.8.193
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab
--principal=oryx/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543,127.12.8.252:35599 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:30:06.770437 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.757496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:60162)
0812 02:30:06.758364 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:06.758392 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:06.758552 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:06.759004 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:06.759009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:06.759560 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:06.759825 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.759837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.760916 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.760920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.762142 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:06.762155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.762889 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.762895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:06.763059 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.763679 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:06.763697 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:06.764542 (+ 845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:06.767419 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.767424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.767429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.767811 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.768168 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:06.768173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:06.768176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:06.768333 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:06.768829 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:06.768833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:06.769018 (+ 185us) client_negotiation.cc:769] Sending connection context
0812 02:30:06.769138 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 02:30:06.769708 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":594,"threads_started":1}
I20260812 02:30:06.771670 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:06.757725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.12.8.254:35939)
0812 02:30:06.757966 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:06.757972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:06.758515 (+ 543us) server_negotiation.cc:407] Connection header received
0812 02:30:06.758668 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:06.758673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:06.758838 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:06.758957 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:06.759968 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.760752 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.762285 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:06.762734 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:06.762844 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:06.764731 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:06.764736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:06.764739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:06.764771 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:06.767288 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.767931 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.767935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.767938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.768063 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:06.768435 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:06.768439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:06.768441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:06.768715 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:06.768852 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:06.771214 (+ 2362us) server_negotiation.cc:299] Negotiation successful
0812 02:30:06.771409 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 02:30:06.772163 19852 tablet_copy_service.cc:140] P c787dfba77e6402dadf24679cfedd84b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ed8f0ac0a76142949e22cd16f7e51fce ({username='oryx', principal='oryx/127.12.8.252@KRBTEST.COM'} at 127.0.0.1:60162)
I20260812 02:30:06.772324 19852 tablet_copy_service.cc:161] P c787dfba77e6402dadf24679cfedd84b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ed8f0ac0a76142949e22cd16f7e51fce at {username='oryx', principal='oryx/127.12.8.252@KRBTEST.COM'} at 127.0.0.1:60162: session id = ed8f0ac0a76142949e22cd16f7e51fce-00000000000000000000000000000000
I20260812 02:30:06.775610 19852 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 02:30:06.777832 20488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 02:30:06.784957 20488 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: tablet copy: Starting download of 0 data blocks...
I20260812 02:30:06.785333 20488 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: tablet copy: Starting download of 1 WAL segments...
I20260812 02:30:06.788205 20488 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 02:30:06.792894 19852 tablet_copy_service.cc:342] P c787dfba77e6402dadf24679cfedd84b: Request end of tablet copy session ed8f0ac0a76142949e22cd16f7e51fce-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.8.252@KRBTEST.COM'} at 127.0.0.1:60162
I20260812 02:30:06.793052 19852 tablet_copy_service.cc:433] P c787dfba77e6402dadf24679cfedd84b: ending tablet copy session ed8f0ac0a76142949e22cd16f7e51fce-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ed8f0ac0a76142949e22cd16f7e51fce
I20260812 02:30:06.807085 20488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:06.810091 20725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:06.810138 20727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:06.819058 20729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:06.828122 20488 server_base.cc:1034] running on GCE node
I20260812 02:30:06.828648 20488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:06.844712 20488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:06.846185 20488 hybrid_clock.cc:648] HybridClock initialized: now 1786501806846120 us; error 70 us; skew 500 ppm
I20260812 02:30:06.846464 20488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:06 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501806, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:06.848830 20488 init.cc:377] Logged in from keytab as oryx/127.12.8.252@KRBTEST.COM (short username oryx)
I20260812 02:30:06.854269 20488 webserver.cc:460] Webserver started at http://127.12.8.252:37615/ using document root <none> and password file <none>
I20260812 02:30:06.854633 20488 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:06.854744 20488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:06.857988 20488 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:30:06.860206 20736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:06.860652 20488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:30:06.860797 20488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
format_stamp: "Formatted at 2026-08-12 02:30:06 on dist-test-slave-vq2q"
server_key: "65550bd8ec958ec23552ae125ebe9f8e"
server_key_iv: "74655b0ec3eb349ab9ef73271f0cdab7"
server_key_version: "encryptionkey@0"
I20260812 02:30:06.861001 20488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:06.900913 20488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:06.904508 20488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:06.939749 20488 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.252:35599
I20260812 02:30:06.939752 20797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.252:35599 every 8 connection(s)
I20260812 02:30:06.940687 20488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
I20260812 02:30:06.946647 20798 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:30:06.949890 20798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Bootstrap starting.
I20260812 02:30:06.962491 20798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:30:06.962954 20798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: Bootstrap complete.
I20260812 02:30:06.963912 20798 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } }
I20260812 02:30:06.964135 20798 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed8f0ac0a76142949e22cd16f7e51fce, State: Initialized, Role: LEARNER
I20260812 02:30:06.964377 20798 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } }
I20260812 02:30:06.965216 20802 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:06.965520 20798 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:06.965665 20802 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: LEARNER
I20260812 02:30:06.973140 20814 catalog_manager.cc:1259] Loaded cluster ID: f2332070953147fab8eb91c78e45377b
I20260812 02:30:06.973256 20814 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: loading cluster ID for follower catalog manager: success
I20260812 02:30:06.976874 20814 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: acquiring CA information for follower catalog manager: success
I20260812 02:30:06.979475 20814 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:30:07.110481 20703 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:07.110895 20703 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:07.123266 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.110652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:58372)
0812 02:30:07.111415 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:07.111434 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:07.111543 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:07.113594 (+ 2051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:07.113599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:07.113633 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:07.114029 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:07.114047 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.118627 (+ 4580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.118632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:07.120388 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:07.120404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.122186 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.122192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:07.122556 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.122585 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:30:07.122671 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 02:30:07.122869 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578}
I20260812 02:30:07.123467 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.111947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.12.8.252:35599)
0812 02:30:07.113111 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:07.113115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:07.113134 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:30:07.113250 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:07.113256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:07.113364 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:07.113495 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:07.114286 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.116395 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.120549 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.122017 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.122138 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.122503 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:07.122710 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 02:30:07.122916 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":669,"threads_started":1}
I20260812 02:30:07.124591 20770 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260812 02:30:07.125074 20802 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c787dfba77e6402dadf24679cfedd84b. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: NON_VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: true } } }
I20260812 02:30:07.125399 20802 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: LEARNER
I20260812 02:30:07.125738 20702 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: attempting to promote NON_VOTER ed8f0ac0a76142949e22cd16f7e51fce to VOTER
I20260812 02:30:07.126585 20702 consensus_queue.cc:237] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } }
I20260812 02:30:07.128786 20770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 LEARNER]: Refusing update from remote peer c787dfba77e6402dadf24679cfedd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 02:30:07.128806 20538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Refusing update from remote peer c787dfba77e6402dadf24679cfedd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 02:30:07.129554 20703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 02:30:07.129920 20699 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 02:30:07.132041 20703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:07.132762 20538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:07.137462 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:07.137997 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:07.138859 20707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:07.139180 20707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:30:07.139729 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:07.140133 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:07.146411 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:07.146773 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:07.147318 20770 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:07.151885 20802 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c787dfba77e6402dadf24679cfedd84b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:07.152263 20802 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: FOLLOWER
W20260812 02:30:07.292868 20724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:07.293303 20724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:07.293512 20724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:07.304836 20724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:07.305071 20724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:07.305341 20724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:30:07.319730 20724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab
--principal=oryx/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:45959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=38951
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543,127.12.8.252:35599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:07.321370 20724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:07.322894 20724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:07.342861 20822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:07.342823 20828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:07.343251 20826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:07.345398 20724 server_base.cc:1034] running on GCE node
I20260812 02:30:07.346292 20724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:07.347671 20724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:07.348902 20724 hybrid_clock.cc:648] HybridClock initialized: now 1786501807348844 us; error 48 us; skew 500 ppm
I20260812 02:30:07.349391 20724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:07 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501807, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:07.354785 20724 init.cc:377] Logged in from keytab as oryx/127.12.8.193@KRBTEST.COM (short username oryx)
I20260812 02:30:07.357154 20724 webserver.cc:460] Webserver started at http://127.12.8.193:38951/ using document root <none> and password file <none>
I20260812 02:30:07.358140 20724 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:07.358371 20724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:07.365784 20724 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.004s
I20260812 02:30:07.370137 20835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:07.371794 20724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:30:07.372059 20724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "572e2683344e422d87faad60b0bcb6d3"
format_stamp: "Formatted at 2026-08-12 02:30:03 on dist-test-slave-vq2q"
server_key: "714cc5b68ad11b36d729ad4263507231"
server_key_iv: "d388bcee997dbe5e3556a6ba045f3b0b"
server_key_version: "encryptionkey@0"
I20260812 02:30:07.373683 20724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:07.403283 20724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:07.409963 20724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:07.410465 20724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:07.412508 20724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:07.415679 20724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:07.415886 20724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:07.416077 20724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:07.416250 20724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:07.496270 20724 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:45959
I20260812 02:30:07.496496 20964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:45959 every 8 connection(s)
I20260812 02:30:07.498229 20724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:30:07.505712 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 20724
I20260812 02:30:07.506285 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 20059
Aug 12 02:30:07 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501807, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.253@KRBTEST.COM
Aug 12 02:30:07 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501807, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
Aug 12 02:30:07 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501807, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.252@KRBTEST.COM
Aug 12 02:30:07 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.194@KRBTEST.COM
I20260812 02:30:07.544137 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.508985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:36711 (local address 127.12.8.253:37543)
0812 02:30:07.510842 (+ 1857us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:07.510848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:07.510871 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:30:07.514186 (+ 3315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:07.514192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:07.514330 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:07.515070 (+ 740us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:07.519057 (+ 3987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.520653 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.525091 (+ 4438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.526143 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.526244 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.533101 (+ 6857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:07.533111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:07.533115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:07.533167 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:07.539150 (+ 5983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:07.540570 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:07.540576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:07.540579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:07.540820 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:07.541505 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:07.541510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:07.541513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:07.542008 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:07.542161 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:07.543241 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0812 02:30:07.543485 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1670,"thread_start_us":1385,"threads_started":1}
I20260812 02:30:07.545032 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.502710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:36461 (local address 127.12.8.254:35939)
0812 02:30:07.503569 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:07.503573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:07.507197 (+ 3624us) server_negotiation.cc:407] Connection header received
0812 02:30:07.514611 (+ 7414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:07.514617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:07.514704 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:07.514823 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:07.518847 (+ 4024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.520403 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.523790 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.524920 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.525035 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.535134 (+ 10099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:07.535144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:07.535149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:07.535199 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:07.540940 (+ 5741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:07.541975 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:07.541980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:07.541984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:07.542195 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:07.542751 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:07.542755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:07.542758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:07.543179 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:07.543320 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:07.543980 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 02:30:07.544610 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":1241,"threads_started":1}
I20260812 02:30:07.546211 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.503179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:36461)
0812 02:30:07.507089 (+ 3910us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:07.507110 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:07.514455 (+ 7345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:07.514893 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:07.514903 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:07.517068 (+ 2165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:07.518629 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:07.518647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.520623 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.520628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:07.523617 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:07.523633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.525591 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.525602 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:07.526357 (+ 755us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.527833 (+ 1476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:07.527879 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:07.534800 (+ 6921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:07.541088 (+ 6288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:07.541092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:07.541095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:07.541828 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:07.542347 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:07.542351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:07.542354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:07.542607 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:07.543396 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:07.543400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:07.543624 (+ 224us) client_negotiation.cc:769] Sending connection context
0812 02:30:07.545251 (+ 1627us) client_negotiation.cc:240] Negotiation successful
0812 02:30:07.545500 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3729,"mutex_wait_us":198,"spinlock_wait_cycles":50560,"thread_start_us":4352,"threads_started":1}
I20260812 02:30:07.547580 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.508221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.12.8.193:36711)
0812 02:30:07.509847 (+ 1626us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:07.509866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:07.513233 (+ 3367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:07.514894 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:07.514913 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:07.517063 (+ 2150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:07.518722 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:07.518734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.520812 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.520817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:07.524916 (+ 4099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:07.524931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.526767 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.526776 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:07.527083 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.527798 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:07.527870 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:07.532762 (+ 4892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:07.539345 (+ 6583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:07.539356 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:07.539374 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:07.540399 (+ 1025us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:07.540971 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:07.540977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:07.540980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:07.541348 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:07.542163 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:07.542172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:07.542735 (+ 563us) client_negotiation.cc:769] Sending connection context
0812 02:30:07.543101 (+ 366us) client_negotiation.cc:240] Negotiation successful
0812 02:30:07.543793 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"mutex_wait_us":141,"spinlock_wait_cycles":30208,"thread_start_us":1286,"threads_started":1}
I20260812 02:30:07.548046 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.502472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:37629 (local address 127.12.8.252:35599)
0812 02:30:07.502709 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:07.502716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:07.506002 (+ 3286us) server_negotiation.cc:407] Connection header received
0812 02:30:07.513581 (+ 7579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:07.513590 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:07.513695 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:07.513805 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:07.518733 (+ 4928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.520244 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.524054 (+ 3810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.524862 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.524987 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.536782 (+ 11795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:07.536792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:07.536796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:07.536851 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:07.542698 (+ 5847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:07.545047 (+ 2349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:07.545054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:07.545057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:07.545341 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:07.546071 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:07.546076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:07.546080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:07.546534 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:07.546663 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:07.547310 (+ 647us) server_negotiation.cc:299] Negotiation successful
0812 02:30:07.547582 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 02:30:07.548285 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.503126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.12.8.193:37629)
0812 02:30:07.505809 (+ 2683us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:07.505867 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:07.513221 (+ 7354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:07.515381 (+ 2160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:07.515387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:07.516667 (+ 1280us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:07.518450 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:07.518483 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.520513 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.520520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:07.523834 (+ 3314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:07.523852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.525591 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.525604 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:07.528714 (+ 3110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.529521 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:07.529548 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:07.536379 (+ 6831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:07.544221 (+ 7842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:07.544226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:07.544230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:07.544894 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:07.545498 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:07.545503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:07.545506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:07.545904 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:07.546769 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:07.546787 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:07.546998 (+ 211us) client_negotiation.cc:769] Sending connection context
0812 02:30:07.547157 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 02:30:07.547621 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2270,"mutex_wait_us":210,"spinlock_wait_cycles":32128,"thread_start_us":1803,"threads_started":1}
I20260812 02:30:07.548442 20966 heartbeater.cc:347] Connected to a master server at 127.12.8.254:35939
I20260812 02:30:07.548724 20966 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:07.549785 20968 heartbeater.cc:347] Connected to a master server at 127.12.8.253:37543
I20260812 02:30:07.549997 20966 heartbeater.cc:511] Master 127.12.8.254:35939 requested a full tablet report, sending...
I20260812 02:30:07.550047 20968 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:07.551075 20967 heartbeater.cc:347] Connected to a master server at 127.12.8.252:35599
I20260812 02:30:07.551245 20968 heartbeater.cc:511] Master 127.12.8.253:37543 requested a full tablet report, sending...
I20260812 02:30:07.551314 20967 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:07.552376 19826 ts_manager.cc:194] Re-registered known tserver with Master: 572e2683344e422d87faad60b0bcb6d3 (127.12.8.193:45959)
I20260812 02:30:07.552593 20967 heartbeater.cc:511] Master 127.12.8.252:35599 requested a full tablet report, sending...
I20260812 02:30:07.553622 19826 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:36461
I20260812 02:30:07.555145 20527 ts_manager.cc:194] Registered new tserver with Master: 572e2683344e422d87faad60b0bcb6d3 (127.12.8.193:45959)
I20260812 02:30:07.555398 20757 ts_manager.cc:194] Registered new tserver with Master: 572e2683344e422d87faad60b0bcb6d3 (127.12.8.193:45959)
I20260812 02:30:07.558173 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:07.508177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.12.8.194:45489)
0812 02:30:07.525161 (+ 16984us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:07.525171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:07.525226 (+ 55us) server_negotiation.cc:407] Connection header received
0812 02:30:07.525305 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:07.525314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:07.525748 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:07.525984 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:07.528727 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.531092 (+ 2365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.533493 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:07.534719 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:07.541817 (+ 7098us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:07.545619 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:07.545641 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:07.545660 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:07.545759 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:07.553394 (+ 7635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:07.554630 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:07.554640 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:07.554644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:07.554950 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:07.555568 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:07.555574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:07.555576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:07.556220 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:07.556655 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:07.557320 (+ 665us) server_negotiation.cc:299] Negotiation successful
0812 02:30:07.557603 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16662,"thread_start_us":18145,"threads_started":1}
I20260812 02:30:07.559623 20205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:07.647838 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 20059
I20260812 02:30:07.669919 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:45489
--local_ip_for_outbound_sockets=127.12.8.194
--tserver_master_addrs=127.12.8.254:35939
--webserver_port=35237
--webserver_interface=127.12.8.194
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab
--principal=oryx/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543,127.12.8.252:35599 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:08.030043 20980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:08.030355 20980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:08.030474 20980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:08.038496 20980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:08.038630 20980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:08.038913 20980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:30:08.049994 20980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab
--principal=oryx/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:45489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=35237
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543,127.12.8.252:35599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:08.051453 20980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:08.052852 20980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:08.068296 20988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:08.068349 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:08.068389 20989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:08.069582 20980 server_base.cc:1034] running on GCE node
I20260812 02:30:08.070421 20980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:08.071694 20980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:08.072966 20980 hybrid_clock.cc:648] HybridClock initialized: now 1786501808072910 us; error 42 us; skew 500 ppm
I20260812 02:30:08.073454 20980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501808, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:08.082927 20980 init.cc:377] Logged in from keytab as oryx/127.12.8.194@KRBTEST.COM (short username oryx)
I20260812 02:30:08.085038 20980 webserver.cc:460] Webserver started at http://127.12.8.194:35237/ using document root <none> and password file <none>
I20260812 02:30:08.086055 20980 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:08.086231 20980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:08.093952 20980 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 02:30:08.098428 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:08.100414 20980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:30:08.100672 20980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81"
format_stamp: "Formatted at 2026-08-12 02:30:04 on dist-test-slave-vq2q"
server_key: "a7e20682c4bd409b09936bf48af754a5"
server_key_iv: "93fd15828d46f09cb85c766a0b99c8fb"
server_key_version: "encryptionkey@0"
I20260812 02:30:08.102066 20980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:08.121622 20980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:08.127548 20980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:08.127923 20980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:08.129325 20980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:08.131285 20980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:08.131409 20980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:08.131548 20980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:08.131640 20980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:08.174508 20980 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:45489
I20260812 02:30:08.174552 21134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:45489 every 8 connection(s)
I20260812 02:30:08.176529 20980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:30:08.180632 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 20980
I20260812 02:30:08.180931 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 20217
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501808, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.253@KRBTEST.COM
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501808, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.252@KRBTEST.COM
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.195@KRBTEST.COM
I20260812 02:30:08.223759 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.180546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:43683 (local address 127.12.8.253:37543)
0812 02:30:08.195988 (+ 15442us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.195997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.197259 (+ 1262us) server_negotiation.cc:407] Connection header received
0812 02:30:08.198755 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.198762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.198860 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.199167 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.201684 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.202932 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.206000 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.206809 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.206938 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.213885 (+ 6947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.213894 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.213899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.213950 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.218793 (+ 4843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.219953 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.219958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.219962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.220196 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.220960 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.220965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.220969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.221471 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.221636 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.222850 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.223153 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15245,"thread_start_us":15433,"threads_started":1}
I20260812 02:30:08.225107 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.181368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.12.8.194:43683)
0812 02:30:08.195898 (+ 14530us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:08.195978 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:08.198378 (+ 2400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:08.199517 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:08.199527 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:08.200246 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:08.201434 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.201465 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.203191 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.203202 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.205772 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.205800 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.207548 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.207558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.208295 (+ 737us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.209482 (+ 1187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:08.209518 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:08.213512 (+ 3994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:08.219044 (+ 5532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.219053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.219068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.219774 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.220357 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.220362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.220366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.220804 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.221724 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:08.221732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:08.222296 (+ 564us) client_negotiation.cc:769] Sending connection context
0812 02:30:08.222718 (+ 422us) client_negotiation.cc:240] Negotiation successful
0812 02:30:08.223380 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14123,"mutex_wait_us":167,"thread_start_us":19061,"threads_started":1}
I20260812 02:30:08.227108 21137 heartbeater.cc:347] Connected to a master server at 127.12.8.253:37543
I20260812 02:30:08.227530 21137 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:08.230491 21137 heartbeater.cc:511] Master 127.12.8.253:37543 requested a full tablet report, sending...
I20260812 02:30:08.233418 20527 ts_manager.cc:194] Registered new tserver with Master: 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489)
I20260812 02:30:08.234056 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.181780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:32999 (local address 127.12.8.252:35599)
0812 02:30:08.197015 (+ 15235us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.197022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.201669 (+ 4647us) server_negotiation.cc:407] Connection header received
0812 02:30:08.201786 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.201793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.201907 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.202258 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.204376 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.205793 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.210527 (+ 4734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.211282 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.211399 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.220196 (+ 8797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.220206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.220211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.220281 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.227263 (+ 6982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.228895 (+ 1632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.228903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.228907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.229250 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.230801 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.230807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.230810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.231416 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.231573 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.233034 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.233416 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15082,"thread_start_us":15237,"threads_started":1}
I20260812 02:30:08.235056 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.181789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.12.8.194:32999)
0812 02:30:08.201037 (+ 19248us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:08.201059 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:08.201478 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:08.202543 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:08.202549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:08.203599 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:08.204125 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.204140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.206028 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.206034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.210302 (+ 4268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.210322 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.211795 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.211801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.212000 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.212930 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:08.212957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:08.217232 (+ 4275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:08.227563 (+ 10331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.227568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.227584 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.228629 (+ 1045us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.229403 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.229409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.229413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.230621 (+ 1208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.231656 (+ 1035us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:08.231661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:08.231897 (+ 236us) client_negotiation.cc:769] Sending connection context
0812 02:30:08.232048 (+ 151us) client_negotiation.cc:240] Negotiation successful
0812 02:30:08.232414 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19097,"mutex_wait_us":2201,"thread_start_us":12466,"threads_started":1}
I20260812 02:30:08.235435 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.183187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.12.8.195:37565)
0812 02:30:08.197811 (+ 14624us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.197822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.199098 (+ 1276us) server_negotiation.cc:407] Connection header received
0812 02:30:08.199225 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.199236 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.199623 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.200097 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.208688 (+ 8591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.209962 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.211996 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.213039 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.213171 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.222832 (+ 9661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.222865 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.222895 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.222958 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.229156 (+ 6198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.230326 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.230339 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.230342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.230696 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.231606 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.231615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.231618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.232398 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.232899 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.234004 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.234719 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14431,"thread_start_us":14245,"threads_started":1}
I20260812 02:30:08.236186 21136 heartbeater.cc:347] Connected to a master server at 127.12.8.252:35599
I20260812 02:30:08.236400 21136 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:08.237259 21136 heartbeater.cc:511] Master 127.12.8.252:35599 requested a full tablet report, sending...
I20260812 02:30:08.237722 20358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:08.238869 20757 ts_manager.cc:194] Registered new tserver with Master: 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489)
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501808, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:08.254695 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.201509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:52745 (local address 127.12.8.254:35939)
0812 02:30:08.216756 (+ 15247us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.216764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.216789 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:30:08.216868 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.216873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.216973 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.217354 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.220094 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.221703 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.237230 (+ 15527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.238910 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.239135 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.245336 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.245346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.245353 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.245426 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.250570 (+ 5144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.251914 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.251918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.251921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.252151 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.252844 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.252848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.252850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.253319 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.253439 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.254047 (+ 608us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.254252 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15049,"thread_start_us":15964,"threads_started":1}
I20260812 02:30:08.254984 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.201370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:52745)
0812 02:30:08.208275 (+ 6905us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:08.208294 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:08.212478 (+ 4184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:08.217513 (+ 5035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:08.217519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:08.218698 (+ 1179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:08.219847 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.219866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.228537 (+ 8671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.228547 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.235942 (+ 7395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.235966 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.239273 (+ 3307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.239282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.239611 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.240529 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:08.240553 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:08.244929 (+ 4376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:08.250845 (+ 5916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.250853 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.250858 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.251716 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.252286 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.252292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.252296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.252658 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.253446 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:08.253450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:08.253705 (+ 255us) client_negotiation.cc:769] Sending connection context
0812 02:30:08.253869 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 02:30:08.254313 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6767,"thread_start_us":6572,"threads_started":1}
I20260812 02:30:08.256362 21135 heartbeater.cc:347] Connected to a master server at 127.12.8.254:35939
I20260812 02:30:08.256623 21135 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:08.257498 21135 heartbeater.cc:511] Master 127.12.8.254:35939 requested a full tablet report, sending...
I20260812 02:30:08.258888 19826 ts_manager.cc:194] Re-registered known tserver with Master: 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489)
I20260812 02:30:08.259865 19826 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:52745
I20260812 02:30:08.340215 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 20217
I20260812 02:30:08.363364 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:37565
--local_ip_for_outbound_sockets=127.12.8.195
--tserver_master_addrs=127.12.8.254:35939
--webserver_port=39923
--webserver_interface=127.12.8.195
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab
--principal=oryx/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543,127.12.8.252:35599 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:30:08.557363 20966 heartbeater.cc:503] Master 127.12.8.254:35939 was elected leader, sending a full tablet report...
W20260812 02:30:08.720538 21149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:08.720846 21149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:08.720968 21149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:08.728545 21149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:08.728674 21149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:08.728823 21149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:08.739760 21149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab
--principal=oryx/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:37565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=39923
--tserver_master_addrs=127.12.8.254:35939,127.12.8.253:37543,127.12.8.252:35599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:08.741251 21149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:08.742601 21149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:08.757916 21155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:08.757830 21156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:08.757870 21159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:08.759330 21149 server_base.cc:1034] running on GCE node
I20260812 02:30:08.760123 21149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:08.761361 21149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:08.762974 21149 hybrid_clock.cc:648] HybridClock initialized: now 1786501808762891 us; error 52 us; skew 500 ppm
I20260812 02:30:08.763492 21149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501808, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:08.767688 21149 init.cc:377] Logged in from keytab as oryx/127.12.8.195@KRBTEST.COM (short username oryx)
I20260812 02:30:08.769759 21149 webserver.cc:460] Webserver started at http://127.12.8.195:39923/ using document root <none> and password file <none>
I20260812 02:30:08.770680 21149 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:08.770824 21149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:08.778352 21149 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260812 02:30:08.782353 21169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:08.784191 21149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:30:08.784425 21149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "78bcbb43058e4f45a71dcb78fc413b2d"
format_stamp: "Formatted at 2026-08-12 02:30:04 on dist-test-slave-vq2q"
server_key: "a5789b787f7584eda3102b9cdd34ebfd"
server_key_iv: "2700c81172cc0d914a806832d0b91011"
server_key_version: "encryptionkey@0"
I20260812 02:30:08.785758 21149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:08.808398 21149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:08.813884 21149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:08.814208 21149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:08.815506 21149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:08.817440 21149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:08.817550 21149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:08.817662 21149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:08.817724 21149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:08.856343 21149 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:37565
I20260812 02:30:08.856377 21302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:37565 every 8 connection(s)
I20260812 02:30:08.858316 21149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:08.868881 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 21149
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501808, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.252@KRBTEST.COM
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501808, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.253@KRBTEST.COM
I20260812 02:30:08.901006 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.862150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:36135 (local address 127.12.8.253:37543)
0812 02:30:08.863351 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.863358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.864229 (+ 871us) server_negotiation.cc:407] Connection header received
0812 02:30:08.866841 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.866848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.866977 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.867125 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.875613 (+ 8488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.876883 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.882852 (+ 5969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.883631 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.883749 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.890861 (+ 7112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.890872 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.890879 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.890945 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.896661 (+ 5716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.897740 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.897746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.897750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.898009 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.898711 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.898716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.898720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.899256 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.899408 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.900020 (+ 612us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.900346 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":853,"threads_started":1}
I20260812 02:30:08.901262 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.862778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.12.8.195:47563)
0812 02:30:08.867471 (+ 4693us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:08.867499 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:08.867640 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:08.868566 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:08.868571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:08.869840 (+ 1269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:08.870847 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.870878 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.873500 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.873507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.876065 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.876087 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.877731 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.877739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.878152 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.879258 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:08.879295 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:08.883148 (+ 3853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:08.891967 (+ 8819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.891981 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.892004 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.893915 (+ 1911us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.894815 (+ 900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.894820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.894824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.895261 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.896267 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:08.896277 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:08.897004 (+ 727us) client_negotiation.cc:769] Sending connection context
0812 02:30:08.897510 (+ 506us) client_negotiation.cc:240] Negotiation successful
0812 02:30:08.898274 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4485,"mutex_wait_us":394,"thread_start_us":363,"threads_started":1}
I20260812 02:30:08.903304 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.862368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:47563 (local address 127.12.8.252:35599)
0812 02:30:08.868098 (+ 5730us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.868104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.868189 (+ 85us) server_negotiation.cc:407] Connection header received
0812 02:30:08.868268 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.868273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.868368 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.868749 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.872032 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.873261 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.876363 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.877199 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.877327 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.883494 (+ 6167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.883503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.883508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.883577 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.891665 (+ 8088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.894191 (+ 2526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.894197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.894200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.894643 (+ 443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.895445 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.895451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.895454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.896053 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.902286 (+ 6233us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.902518 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.902814 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5582,"thread_start_us":7987,"threads_started":1}
I20260812 02:30:08.905159 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.862783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.12.8.195:36135)
0812 02:30:08.864015 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:08.864064 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:08.866489 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:08.867717 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:08.867731 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:08.871487 (+ 3756us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:08.871806 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.871819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.880938 (+ 9119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.880945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.882595 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.882621 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.884295 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.884304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.884570 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.885460 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:08.885488 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:08.890416 (+ 4928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:08.896840 (+ 6424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.896845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.896850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.897602 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.898184 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.898189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.898193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.898569 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.899491 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:08.899496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:08.899668 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 02:30:08.903920 (+ 4252us) client_negotiation.cc:240] Negotiation successful
0812 02:30:08.904407 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":4177,"threads_started":1}
I20260812 02:30:08.905323 21304 heartbeater.cc:347] Connected to a master server at 127.12.8.252:35599
I20260812 02:30:08.905900 21304 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:08.907025 21305 heartbeater.cc:347] Connected to a master server at 127.12.8.253:37543
I20260812 02:30:08.907874 21305 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:08.908603 21304 heartbeater.cc:511] Master 127.12.8.252:35599 requested a full tablet report, sending...
I20260812 02:30:08.910988 21305 heartbeater.cc:511] Master 127.12.8.253:37543 requested a full tablet report, sending...
I20260812 02:30:08.911402 20757 ts_manager.cc:194] Registered new tserver with Master: 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565)
I20260812 02:30:08.913440 20527 ts_manager.cc:194] Registered new tserver with Master: 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565)
Aug 12 02:30:08 dist-test-slave-vq2q krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501808, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:08.935933 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.867991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:58763 (local address 127.12.8.254:35939)
0812 02:30:08.880553 (+ 12562us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.880561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.885258 (+ 4697us) server_negotiation.cc:407] Connection header received
0812 02:30:08.885363 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.885370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.885483 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.885622 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.910171 (+ 24549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.911906 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.915004 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.915895 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.916040 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.924697 (+ 8657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.924705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.924711 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.924759 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.930759 (+ 6000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.932079 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.932085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.932096 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.932319 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.933040 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.933050 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.933053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.933624 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.933801 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.935026 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.935398 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12384,"thread_start_us":12581,"threads_started":1}
I20260812 02:30:08.936252 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.867907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:58763)
0812 02:30:08.884818 (+ 16911us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:08.884870 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:08.885077 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:08.891670 (+ 6593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:08.891676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:08.906207 (+ 14531us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:08.909903 (+ 3696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.909926 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.912235 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.912243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.914789 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:08.914812 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.916334 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.916344 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:08.916693 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.917965 (+ 1272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:08.918000 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:08.924339 (+ 6339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:08.931181 (+ 6842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.931188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.931193 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.931927 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.932446 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:08.932452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:08.932455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:08.932850 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:08.933773 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:08.933778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:08.934020 (+ 242us) client_negotiation.cc:769] Sending connection context
0812 02:30:08.934750 (+ 730us) client_negotiation.cc:240] Negotiation successful
0812 02:30:08.935597 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16668,"thread_start_us":14040,"threads_started":1}
I20260812 02:30:08.938273 21303 heartbeater.cc:347] Connected to a master server at 127.12.8.254:35939
I20260812 02:30:08.938587 21303 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:08.939800 21303 heartbeater.cc:511] Master 127.12.8.254:35939 requested a full tablet report, sending...
I20260812 02:30:08.941586 19826 ts_manager.cc:194] Re-registered known tserver with Master: 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565)
I20260812 02:30:08.943223 19826 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:58763
I20260812 02:30:08.947834 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.893119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.12.8.254:35939)
0812 02:30:08.898766 (+ 5647us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.898773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.913977 (+ 15204us) server_negotiation.cc:407] Connection header received
0812 02:30:08.914313 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.914319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.914422 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.914539 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.921517 (+ 6978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.923026 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.925874 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.926931 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.927092 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.937651 (+ 10559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.937659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.937665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.937712 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.942862 (+ 5150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.943730 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.943734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.943737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.943957 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.945297 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.945303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.945306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.945822 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.946184 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.946921 (+ 737us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.947202 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5438,"thread_start_us":5177,"threads_started":1}
I20260812 02:30:08.956271 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.885617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.12.8.252:35599)
0812 02:30:08.896339 (+ 10722us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.896347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.903754 (+ 7407us) server_negotiation.cc:407] Connection header received
0812 02:30:08.904425 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.904434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.904547 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.904644 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.919914 (+ 15270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.921909 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.925048 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.926137 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.926265 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.933485 (+ 7220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.933499 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.933505 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.933564 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.938957 (+ 5393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.952888 (+ 13931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.952897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.952901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.953304 (+ 403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.953908 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.953914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.953917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.954404 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.954870 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.955228 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.955632 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10409,"thread_start_us":16330,"threads_started":1}
I20260812 02:30:08.957012 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.916104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.12.8.253:37543)
0812 02:30:08.916377 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.916383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:08.918978 (+ 2595us) server_negotiation.cc:407] Connection header received
0812 02:30:08.919285 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:08.919292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:08.919421 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:08.919514 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:08.926741 (+ 7227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.928326 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.931651 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:08.932680 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:08.932790 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:08.946475 (+ 13685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:08.946487 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:08.946492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:08.946548 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:08.951935 (+ 5387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.953321 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.953327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.953331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.953533 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:08.954053 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:08.954057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:08.954060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:08.954606 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:08.954994 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:08.955978 (+ 984us) server_negotiation.cc:299] Negotiation successful
0812 02:30:08.956339 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 02:30:08.954559 19826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:30:08.967347 19826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:30:09.024930 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.997232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.12.8.195:37565)
0812 02:30:08.999321 (+ 2089us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:08.999464 (+ 143us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.007066 (+ 7602us) server_negotiation.cc:407] Connection header received
0812 02:30:09.007240 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.007250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.007667 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.009769 (+ 2102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:09.012412 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.015847 (+ 3435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.019944 (+ 4097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.022067 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.022215 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.022520 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.023942 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.024170 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1785,"thread_start_us":1477,"threads_started":1}
I20260812 02:30:09.024943 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.996892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:37565 (local address 127.0.0.1:44850)
0812 02:30:09.004725 (+ 7833us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:09.004789 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:09.005251 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:09.009746 (+ 4495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:09.009753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:09.009788 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:09.012043 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.012080 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.016063 (+ 3983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.016069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.019763 (+ 3694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.019778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.023506 (+ 3728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.023512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.023722 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.023760 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 02:30:09.023857 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 02:30:09.024025 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":1,"thread_start_us":949,"threads_started":1}
I20260812 02:30:09.028227 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.998389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:45489 (local address 127.0.0.1:33254)
0812 02:30:09.014855 (+ 16466us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:09.014880 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:09.015391 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:09.017415 (+ 2024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:09.017421 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:09.017506 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:09.018647 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.018661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.020723 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.020728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.024214 (+ 3486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.024232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.026647 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.026658 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.026908 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.026948 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 02:30:09.027065 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 02:30:09.027242 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15785,"mutex_wait_us":27,"thread_start_us":15350,"threads_started":1}
I20260812 02:30:09.032294 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.999808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:45959 (local address 127.0.0.1:43598)
0812 02:30:09.015933 (+ 16125us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:09.015956 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:09.016412 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:09.017999 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:09.018006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:09.018046 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:09.019497 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.019517 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.025323 (+ 5806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.025329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.028312 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.028328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.031269 (+ 2941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.031274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.031485 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.031513 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:30:09.031596 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:30:09.031742 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15938,"thread_start_us":18127,"threads_started":1}
I20260812 02:30:09.032878 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.000385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.12.8.193:45959)
0812 02:30:09.012918 (+ 12533us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.012930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.016305 (+ 3375us) server_negotiation.cc:407] Connection header received
0812 02:30:09.016633 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.016644 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.017018 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.017533 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:09.019814 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.023214 (+ 3400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.028585 (+ 5371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.030549 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.030738 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.030959 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.031788 (+ 829us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.032138 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12248,"thread_start_us":14228,"threads_started":1}
I20260812 02:30:09.032414 21225 tablet_service.cc:1467] Processing CreateTablet for tablet fadd74b6d22247f388c02edf7c91c2da (DEFAULT_TABLE table=test-table [id=3e2b2d5f14b94746841dbbfb24dda11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:09.034794 21225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadd74b6d22247f388c02edf7c91c2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:09.038148 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:08.998755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.12.8.194:45489)
0812 02:30:09.014967 (+ 16212us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.014981 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.015286 (+ 305us) server_negotiation.cc:407] Connection header received
0812 02:30:09.016693 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.016701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.016965 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.017477 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:09.019007 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.020531 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.024441 (+ 3910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.026165 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.026305 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.026649 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.037532 (+ 10883us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.037770 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15892,"thread_start_us":16188,"threads_started":1}
I20260812 02:30:09.039671 20891 tablet_service.cc:1467] Processing CreateTablet for tablet fadd74b6d22247f388c02edf7c91c2da (DEFAULT_TABLE table=test-table [id=3e2b2d5f14b94746841dbbfb24dda11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:09.041970 20891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadd74b6d22247f388c02edf7c91c2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:09.043771 21057 tablet_service.cc:1467] Processing CreateTablet for tablet fadd74b6d22247f388c02edf7c91c2da (DEFAULT_TABLE table=test-table [id=3e2b2d5f14b94746841dbbfb24dda11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:09.045604 21057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadd74b6d22247f388c02edf7c91c2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:09.057879 21347 tablet_bootstrap.cc:492] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Bootstrap starting.
I20260812 02:30:09.062712 21347 tablet_bootstrap.cc:654] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:09.064077 21348 tablet_bootstrap.cc:492] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Bootstrap starting.
I20260812 02:30:09.066823 21347 log.cc:824] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:09.067852 21349 tablet_bootstrap.cc:492] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Bootstrap starting.
I20260812 02:30:09.070205 21348 tablet_bootstrap.cc:654] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:09.072633 21348 log.cc:824] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:09.073068 21347 tablet_bootstrap.cc:492] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: No bootstrap required, opened a new log
I20260812 02:30:09.073457 21347 ts_tablet_manager.cc:1397] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260812 02:30:09.074520 21349 tablet_bootstrap.cc:654] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:09.077059 21349 log.cc:824] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:09.078279 21348 tablet_bootstrap.cc:492] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: No bootstrap required, opened a new log
I20260812 02:30:09.078848 21348 ts_tablet_manager.cc:1397] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260812 02:30:09.084367 21349 tablet_bootstrap.cc:492] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: No bootstrap required, opened a new log
I20260812 02:30:09.084905 21349 ts_tablet_manager.cc:1397] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20260812 02:30:09.087682 21347 raft_consensus.cc:348] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:09.088330 21347 raft_consensus.cc:374] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:09.088505 21347 raft_consensus.cc:729] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bcbb43058e4f45a71dcb78fc413b2d, State: Initialized, Role: FOLLOWER
I20260812 02:30:09.089315 21347 consensus_queue.cc:260] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:09.091893 21303 heartbeater.cc:503] Master 127.12.8.254:35939 was elected leader, sending a full tablet report...
I20260812 02:30:09.092475 21347 ts_tablet_manager.cc:1428] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Time spent starting tablet: real 0.019s user 0.009s sys 0.008s
I20260812 02:30:09.100225 21348 raft_consensus.cc:348] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:09.101140 21348 raft_consensus.cc:374] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:09.101436 21348 raft_consensus.cc:729] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572e2683344e422d87faad60b0bcb6d3, State: Initialized, Role: FOLLOWER
I20260812 02:30:09.102535 21348 consensus_queue.cc:260] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:09.105945 21348 ts_tablet_manager.cc:1428] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Time spent starting tablet: real 0.027s user 0.016s sys 0.009s
I20260812 02:30:09.110321 21349 raft_consensus.cc:348] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:09.111115 21349 raft_consensus.cc:374] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:09.111286 21349 raft_consensus.cc:729] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab3e5e9b4234cbdb36294fcbe6eac81, State: Initialized, Role: FOLLOWER
I20260812 02:30:09.112298 21349 consensus_queue.cc:260] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
W20260812 02:30:09.112560 21307 tablet.cc:2369] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:30:09.114349 21135 heartbeater.cc:503] Master 127.12.8.254:35939 was elected leader, sending a full tablet report...
I20260812 02:30:09.115387 21349 ts_tablet_manager.cc:1428] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Time spent starting tablet: real 0.030s user 0.010s sys 0.016s
W20260812 02:30:09.181115 21138 tablet.cc:2369] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:30:09.266943 20969 tablet.cc:2369] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:30:09.437096 21358 raft_consensus.cc:482] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:09.437717 21358 raft_consensus.cc:504] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:09.440690 21358 leader_election.cc:288] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565), 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489)
I20260812 02:30:09.450956 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.441026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:59387 (local address 127.12.8.195:37565)
0812 02:30:09.441373 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.441379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.442054 (+ 675us) server_negotiation.cc:407] Connection header received
0812 02:30:09.442289 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.442296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.442405 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.442660 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:09.443642 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.445060 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.448145 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.449679 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.449906 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.450013 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.450495 (+ 482us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.450692 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260812 02:30:09.451367 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.440914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:37565 (local address 127.12.8.193:59387)
0812 02:30:09.441892 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:09.441919 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:09.442092 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:09.442849 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:09.442856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:09.442890 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:09.443473 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.443485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.445282 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.445286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.447917 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.447932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.450061 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.450067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.450280 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.450308 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:30:09.450402 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 02:30:09.450540 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":29,"thread_start_us":603,"threads_started":1}
I20260812 02:30:09.452430 21246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadd74b6d22247f388c02edf7c91c2da" candidate_uuid: "572e2683344e422d87faad60b0bcb6d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" is_pre_election: true
I20260812 02:30:09.452899 21246 raft_consensus.cc:2454] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572e2683344e422d87faad60b0bcb6d3 in term 0.
I20260812 02:30:09.453586 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.442324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:46659 (local address 127.12.8.194:45489)
0812 02:30:09.442629 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.442636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.443700 (+ 1064us) server_negotiation.cc:407] Connection header received
0812 02:30:09.443965 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.443969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.444036 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.444109 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:09.445481 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.446557 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.450422 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.452133 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.452342 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.452438 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.452947 (+ 509us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.453111 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 02:30:09.453552 20837 leader_election.cc:302] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572e2683344e422d87faad60b0bcb6d3, 78bcbb43058e4f45a71dcb78fc413b2d; no voters:
I20260812 02:30:09.453912 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.441971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:45489 (local address 127.12.8.193:46659)
0812 02:30:09.443577 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:09.443601 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:09.443793 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:09.444155 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:09.444161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:09.444191 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:09.445142 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.445158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.446765 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.446769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.450239 (+ 3470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:09.450256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.452528 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.452535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:09.452767 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.452799 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:30:09.452895 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 02:30:09.453058 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"mutex_wait_us":1,"thread_start_us":1129,"threads_started":1}
I20260812 02:30:09.454057 21358 raft_consensus.cc:2781] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:30:09.454465 21358 raft_consensus.cc:482] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:09.454655 21358 raft_consensus.cc:3037] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:09.455680 21081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadd74b6d22247f388c02edf7c91c2da" candidate_uuid: "572e2683344e422d87faad60b0bcb6d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" is_pre_election: true
I20260812 02:30:09.456651 21081 raft_consensus.cc:2454] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572e2683344e422d87faad60b0bcb6d3 in term 0.
I20260812 02:30:09.458823 21358 raft_consensus.cc:504] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:09.459764 21358 leader_election.cc:288] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [CANDIDATE]: Term 1 election: Requested vote from peers 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565), 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489)
I20260812 02:30:09.460310 21081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadd74b6d22247f388c02edf7c91c2da" candidate_uuid: "572e2683344e422d87faad60b0bcb6d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81"
I20260812 02:30:09.460407 21246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadd74b6d22247f388c02edf7c91c2da" candidate_uuid: "572e2683344e422d87faad60b0bcb6d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bcbb43058e4f45a71dcb78fc413b2d"
I20260812 02:30:09.460649 21081 raft_consensus.cc:3037] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:09.460716 21246 raft_consensus.cc:3037] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:09.465884 21246 raft_consensus.cc:2454] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572e2683344e422d87faad60b0bcb6d3 in term 1.
I20260812 02:30:09.465884 21081 raft_consensus.cc:2454] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572e2683344e422d87faad60b0bcb6d3 in term 1.
I20260812 02:30:09.466598 20837 leader_election.cc:302] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572e2683344e422d87faad60b0bcb6d3, 78bcbb43058e4f45a71dcb78fc413b2d; no voters:
I20260812 02:30:09.467077 21358 raft_consensus.cc:2781] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:09.467952 21358 raft_consensus.cc:686] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 1 LEADER]: Becoming Leader. State: Replica: 572e2683344e422d87faad60b0bcb6d3, State: Running, Role: LEADER
I20260812 02:30:09.468864 21358 consensus_queue.cc:237] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:09.475124 19826 catalog_manager.cc:5495] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572e2683344e422d87faad60b0bcb6d3 (127.12.8.193). New cstate: current_term: 1 leader_uuid: "572e2683344e422d87faad60b0bcb6d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } health_report { overall_health: HEALTHY } } }
I20260812 02:30:09.588550 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.580145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.12.8.193:45959)
0812 02:30:09.580742 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.580748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.581116 (+ 368us) server_negotiation.cc:407] Connection header received
0812 02:30:09.581395 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.581398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.581490 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.581716 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:09.582436 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.583480 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.585531 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.586352 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.586462 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.587180 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:09.587788 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:09.587951 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.588055 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.588135 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":297,"threads_started":1}
I20260812 02:30:09.601644 21081 raft_consensus.cc:1261] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 1 FOLLOWER]: Refusing update from remote peer 572e2683344e422d87faad60b0bcb6d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:09.601816 21246 raft_consensus.cc:1261] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 1 FOLLOWER]: Refusing update from remote peer 572e2683344e422d87faad60b0bcb6d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:09.603031 21363 consensus_queue.cc:1035] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 02:30:09.603768 21358 consensus_queue.cc:1035] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:09.622328 21370 mvcc.cc:205] Tried to move back new op lower bound from 7317511412113620992 to 7317511411590344704. Current Snapshot: MvccSnapshot[applied={T|T < 7317511412113620992}]
I20260812 02:30:09.624238 21372 mvcc.cc:205] Tried to move back new op lower bound from 7317511412113620992 to 7317511411590344704. Current Snapshot: MvccSnapshot[applied={T|T < 7317511412113620992}]
I20260812 02:30:09.649394 19825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:09.649905 19825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:09.659775 19825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c787dfba77e6402dadf24679cfedd84b: Sending DeleteTablet for 3 replicas of tablet fadd74b6d22247f388c02edf7c91c2da
I20260812 02:30:09.661950 21225 tablet_service.cc:1514] Processing DeleteTablet for tablet fadd74b6d22247f388c02edf7c91c2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:09 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:44850
I20260812 02:30:09.662482 21057 tablet_service.cc:1514] Processing DeleteTablet for tablet fadd74b6d22247f388c02edf7c91c2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:09 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33254
I20260812 02:30:09.663991 20891 tablet_service.cc:1514] Processing DeleteTablet for tablet fadd74b6d22247f388c02edf7c91c2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:09 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:43598
I20260812 02:30:09.674098 21384 tablet_replica.cc:330] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: stopping tablet replica
I20260812 02:30:09.674983 21384 raft_consensus.cc:2229] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:09.675999 21384 raft_consensus.cc:2258] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:09.684455 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 19787
I20260812 02:30:09.685946 21385 tablet_replica.cc:330] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: stopping tablet replica
I20260812 02:30:09.686034 19862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:09.686812 21385 raft_consensus.cc:2229] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:09.688866 21385 raft_consensus.cc:2258] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:09.689759 21384 ts_tablet_manager.cc:1905] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:09.690013 21386 tablet_replica.cc:330] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: stopping tablet replica
I20260812 02:30:09.692224 21385 ts_tablet_manager.cc:1905] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:09.694481 21386 raft_consensus.cc:2229] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:30:09.699105 21384 ts_tablet_manager.cc:1918] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:09.695426 21386 raft_consensus.cc:2258] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:09.699497 21384 log.cc:1196] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/fadd74b6d22247f388c02edf7c91c2da
I20260812 02:30:09.700701 21384 ts_tablet_manager.cc:1939] T fadd74b6d22247f388c02edf7c91c2da P 78bcbb43058e4f45a71dcb78fc413b2d: Deleting consensus metadata
I20260812 02:30:09.702666 21386 ts_tablet_manager.cc:1905] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:09.705502 19809 catalog_manager.cc:4841] TS 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565): tablet fadd74b6d22247f388c02edf7c91c2da (table test-table [id=3e2b2d5f14b94746841dbbfb24dda11d]) successfully deleted
I20260812 02:30:09.704977 21385 ts_tablet_manager.cc:1918] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:09.707538 21385 log.cc:1196] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/fadd74b6d22247f388c02edf7c91c2da
I20260812 02:30:09.708498 21385 ts_tablet_manager.cc:1939] T fadd74b6d22247f388c02edf7c91c2da P 0ab3e5e9b4234cbdb36294fcbe6eac81: Deleting consensus metadata
I20260812 02:30:09.716806 21386 ts_tablet_manager.cc:1918] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:09.717666 21386 log.cc:1196] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/fadd74b6d22247f388c02edf7c91c2da
I20260812 02:30:09.718789 21386 ts_tablet_manager.cc:1939] T fadd74b6d22247f388c02edf7c91c2da P 572e2683344e422d87faad60b0bcb6d3: Deleting consensus metadata
I20260812 02:30:09.723938 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.666790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.12.8.253:37543)
0812 02:30:09.677513 (+ 10723us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.677522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.679222 (+ 1700us) server_negotiation.cc:407] Connection header received
0812 02:30:09.680017 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.680024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.680131 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.680455 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:09.687920 (+ 7465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.689413 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.698015 (+ 8602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.698977 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.699146 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.709371 (+ 10225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:09.709381 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:09.709386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:09.709470 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:09.716299 (+ 6829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:09.717995 (+ 1696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:09.718004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:09.718009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:09.718344 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:09.719019 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:09.719026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:09.719030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:09.719880 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:09.720450 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.722687 (+ 2237us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.723105 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10482,"thread_start_us":10312,"threads_started":1}
I20260812 02:30:09.755596 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.679136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.12.8.252:35599)
0812 02:30:09.681421 (+ 2285us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.681427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.724934 (+ 43507us) server_negotiation.cc:407] Connection header received
0812 02:30:09.725297 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.725303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.725443 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.725541 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:09.728791 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.730905 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.733926 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.735132 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.735349 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.742312 (+ 6963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:09.742325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:09.742332 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:09.742413 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:09.749482 (+ 7069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:09.751305 (+ 1823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:09.751313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:09.751319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:09.751668 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:09.752346 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:09.752352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:09.752356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:09.753150 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:09.753910 (+ 760us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.754203 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.754783 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2119,"thread_start_us":1904,"threads_started":1}
I20260812 02:30:09.867000 19809 catalog_manager.cc:4841] TS 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489): tablet fadd74b6d22247f388c02edf7c91c2da (table test-table [id=3e2b2d5f14b94746841dbbfb24dda11d]) successfully deleted
I20260812 02:30:09.867489 19811 catalog_manager.cc:4841] TS 572e2683344e422d87faad60b0bcb6d3 (127.12.8.193:45959): tablet fadd74b6d22247f388c02edf7c91c2da (table test-table [id=3e2b2d5f14b94746841dbbfb24dda11d]) successfully deleted
I20260812 02:30:09.867861 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 19787
I20260812 02:30:09.931643 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.906686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.12.8.252:35599)
0812 02:30:09.906945 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.906953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.909351 (+ 2398us) server_negotiation.cc:407] Connection header received
0812 02:30:09.909457 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.909464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.909607 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.909735 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:09.913542 (+ 3807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.914878 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.917403 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.918195 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.918331 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.924297 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:09.924309 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:09.924315 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:09.924388 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:09.928112 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:09.928812 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:09.928818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:09.928820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:09.928969 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:09.929688 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:09.929693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:09.929697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:09.930125 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:09.930415 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.930982 (+ 567us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.931263 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 02:30:09.935901 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:09.908504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.12.8.253:37543)
0812 02:30:09.909284 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:09.909292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:09.915416 (+ 6124us) server_negotiation.cc:407] Connection header received
0812 02:30:09.916196 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:09.916204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:09.916329 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:09.916412 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:09.919836 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.921097 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.923532 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:09.924221 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:09.924359 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:09.927822 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:09.927832 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:09.927839 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:09.927908 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:09.932138 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:09.933122 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:09.933131 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:09.933135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:09.933433 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:09.933938 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:09.933944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:09.933947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:09.934429 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:09.934798 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:09.935103 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 02:30:09.935443 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548}
I20260812 02:30:10.641021 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:10.639839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:37669)
0812 02:30:10.640534 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:10.640714 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":355,"threads_started":1}
I20260812 02:30:10.641307 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:10.640202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:47975)
0812 02:30:10.640817 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:10.641010 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":337,"threads_started":1}
W20260812 02:30:10.642097 21135 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:35939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
W20260812 02:30:10.642189 20966 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:35939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:10.642930 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:10.641458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:60883)
0812 02:30:10.642561 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:10.642713 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":285,"threads_started":1}
W20260812 02:30:10.643359 21303 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:35939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:11.159804 21405 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c787dfba77e6402dadf24679cfedd84b)
I20260812 02:30:11.160064 21405 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } }
I20260812 02:30:11.160522 21406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c787dfba77e6402dadf24679cfedd84b)
I20260812 02:30:11.160804 21406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } }
I20260812 02:30:11.161119 21405 leader_election.cc:288] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939), 34598cea9ca64d5ea37ded12f60125cf (127.12.8.253:37543)
I20260812 02:30:11.162737 21406 leader_election.cc:288] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939), ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252:35599)
I20260812 02:30:11.164153 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.162849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:45710)
0812 02:30:11.163506 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.163766 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":665,"threads_started":1}
W20260812 02:30:11.164575 20512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:11.165109 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.161423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:45698)
0812 02:30:11.164522 (+ 3099us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.164747 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2964,"thread_start_us":2827,"threads_started":1}
W20260812 02:30:11.167307 20740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:11.168040 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.167579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:45726)
0812 02:30:11.167732 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.167836 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 02:30:11.168305 20512 leader_election.cc:334] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939): Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:11.172148 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.171401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:45730)
0812 02:30:11.171609 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.171843 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 02:30:11.172399 20740 leader_election.cc:334] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939): Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:11.175104 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.164707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:37543 (local address 127.0.0.1:36418)
0812 02:30:11.165430 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.165452 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:11.165708 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:11.168262 (+ 2554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:11.168268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:11.168324 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:11.168645 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:11.168656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:11.170705 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:11.170710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:11.172302 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:11.172317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:11.174185 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:11.174192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:11.174397 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:11.174430 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 02:30:11.174551 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 02:30:11.174728 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":1299,"threads_started":1}
I20260812 02:30:11.175201 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.166240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.12.8.252:35599)
0812 02:30:11.167810 (+ 1570us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:11.167817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:11.167839 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:30:11.167917 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:11.167923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:11.168011 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:11.168830 (+ 819us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:11.169080 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:11.170156 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:11.172602 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:11.173981 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:11.174102 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:11.174217 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:11.174701 (+ 484us) server_negotiation.cc:299] Negotiation successful
0812 02:30:11.174854 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":700,"threads_started":1}
I20260812 02:30:11.175235 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.163369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:43004)
0812 02:30:11.164123 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.164155 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:11.166254 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:11.168199 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:11.168204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:11.168234 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:11.168651 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:11.168667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:11.170333 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:11.170340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:11.172423 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:11.172439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:11.174315 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:11.174321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:11.174462 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:11.174486 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:30:11.174586 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 02:30:11.174784 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":1,"thread_start_us":480,"threads_started":1}
I20260812 02:30:11.175846 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.164801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.12.8.253:37543)
0812 02:30:11.167837 (+ 3036us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:11.167844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:11.167867 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:30:11.167941 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:11.167946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:11.168092 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:11.168798 (+ 706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:11.169051 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:11.170509 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:11.172459 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:11.173890 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:11.174055 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:11.174195 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:11.174594 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 02:30:11.175475 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":714,"server-negotiator.queue_time_us":2884,"thread_start_us":2657,"threads_started":1}
I20260812 02:30:11.176231 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34598cea9ca64d5ea37ded12f60125cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" is_pre_election: true
I20260812 02:30:11.176491 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "34598cea9ca64d5ea37ded12f60125cf" is_pre_election: true
I20260812 02:30:11.176668 20770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34598cea9ca64d5ea37ded12f60125cf in term 1.
I20260812 02:30:11.176831 20538 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed8f0ac0a76142949e22cd16f7e51fce in term 1.
I20260812 02:30:11.177268 20510 leader_election.cc:302] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34598cea9ca64d5ea37ded12f60125cf, ed8f0ac0a76142949e22cd16f7e51fce; no voters: c787dfba77e6402dadf24679cfedd84b
I20260812 02:30:11.177754 21406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 02:30:11.177752 20740 leader_election.cc:302] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34598cea9ca64d5ea37ded12f60125cf, ed8f0ac0a76142949e22cd16f7e51fce; no voters: c787dfba77e6402dadf24679cfedd84b
I20260812 02:30:11.177889 21406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader c787dfba77e6402dadf24679cfedd84b)
I20260812 02:30:11.177992 21406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:30:11.178258 21405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 02:30:11.178400 21405 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 FOLLOWER]: Starting leader election (detected failure of leader c787dfba77e6402dadf24679cfedd84b)
I20260812 02:30:11.178515 21405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:30:11.180902 21406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } }
I20260812 02:30:11.181093 21405 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } }
I20260812 02:30:11.181787 21406 leader_election.cc:288] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 2 election: Requested vote from peers c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939), ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252:35599)
I20260812 02:30:11.181810 21405 leader_election.cc:288] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [CANDIDATE]: Term 2 election: Requested vote from peers c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939), 34598cea9ca64d5ea37ded12f60125cf (127.12.8.253:37543)
I20260812 02:30:11.182504 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.181981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:45750)
0812 02:30:11.182167 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.182282 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 02:30:11.182542 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.181981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:45746)
0812 02:30:11.182231 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.182356 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260812 02:30:11.182650 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "34598cea9ca64d5ea37ded12f60125cf"
I20260812 02:30:11.183008 20538 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ed8f0ac0a76142949e22cd16f7e51fce in current term 2: Already voted for candidate 34598cea9ca64d5ea37ded12f60125cf in this term.
I20260812 02:30:11.183136 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34598cea9ca64d5ea37ded12f60125cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
I20260812 02:30:11.183482 20770 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 34598cea9ca64d5ea37ded12f60125cf in current term 2: Already voted for candidate ed8f0ac0a76142949e22cd16f7e51fce in this term.
I20260812 02:30:11.183807 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.183297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:45758)
0812 02:30:11.183536 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.183640 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 02:30:11.184084 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.183306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:45752)
0812 02:30:11.183850 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.183945 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
W20260812 02:30:11.184145 20740 leader_election.cc:334] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939): Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
W20260812 02:30:11.184278 20512 leader_election.cc:334] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939): Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:11.184306 20740 leader_election.cc:302] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed8f0ac0a76142949e22cd16f7e51fce; no voters: 34598cea9ca64d5ea37ded12f60125cf, c787dfba77e6402dadf24679cfedd84b
I20260812 02:30:11.184471 20512 leader_election.cc:302] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34598cea9ca64d5ea37ded12f60125cf; no voters: c787dfba77e6402dadf24679cfedd84b, ed8f0ac0a76142949e22cd16f7e51fce
I20260812 02:30:11.184633 21405 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 02:30:11.184762 21406 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 02:30:11.644191 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.643029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:49521)
0812 02:30:11.643769 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.643925 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":440,"threads_started":1}
I20260812 02:30:11.644624 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.643014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:41907)
0812 02:30:11.644281 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.644390 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1142,"thread_start_us":1192,"threads_started":1}
I20260812 02:30:11.645852 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:11.644100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:33675)
0812 02:30:11.645549 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:11.645641 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1334,"thread_start_us":1230,"threads_started":1}
I20260812 02:30:12.646804 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:12.645647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:40383)
0812 02:30:12.646385 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:12.646523 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":434,"threads_started":1}
I20260812 02:30:12.648124 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:12.645758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:36433)
0812 02:30:12.647743 (+ 1985us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:12.647864 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1839,"thread_start_us":1709,"threads_started":1}
I20260812 02:30:12.648195 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:12.646719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:55623)
0812 02:30:12.647865 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:12.647956 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1011,"thread_start_us":944,"threads_started":1}
I20260812 02:30:13.045449 21422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:13.045771 21422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } }
I20260812 02:30:13.046587 21422 leader_election.cc:288] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939), ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252:35599)
I20260812 02:30:13.047451 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34598cea9ca64d5ea37ded12f60125cf" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" is_pre_election: true
I20260812 02:30:13.047792 20770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34598cea9ca64d5ea37ded12f60125cf in term 2.
I20260812 02:30:13.047963 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.046871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:46012)
0812 02:30:13.047495 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.047684 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":326,"threads_started":1}
I20260812 02:30:13.048382 20510 leader_election.cc:302] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34598cea9ca64d5ea37ded12f60125cf, ed8f0ac0a76142949e22cd16f7e51fce; no voters:
I20260812 02:30:13.048828 21422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 02:30:13.048980 21422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:13.049129 21422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:30:13.049849 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.049346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:46026)
0812 02:30:13.049568 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.049668 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 02:30:13.050086 20512 leader_election.cc:334] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939): Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:13.051906 21422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } }
I20260812 02:30:13.052683 21422 leader_election.cc:288] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 3 election: Requested vote from peers c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939), ed8f0ac0a76142949e22cd16f7e51fce (127.12.8.252:35599)
I20260812 02:30:13.053421 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.052886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:46036)
0812 02:30:13.053106 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.053238 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 02:30:13.053396 20770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34598cea9ca64d5ea37ded12f60125cf" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed8f0ac0a76142949e22cd16f7e51fce"
I20260812 02:30:13.053719 20770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:30:13.054435 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.054068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:46038)
0812 02:30:13.054223 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.054302 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 02:30:13.054629 20512 leader_election.cc:334] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939): Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
I20260812 02:30:13.056365 20770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34598cea9ca64d5ea37ded12f60125cf in term 3.
I20260812 02:30:13.056908 20510 leader_election.cc:302] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34598cea9ca64d5ea37ded12f60125cf, ed8f0ac0a76142949e22cd16f7e51fce; no voters: c787dfba77e6402dadf24679cfedd84b
I20260812 02:30:13.057271 21422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:30:13.057747 21422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [term 3 LEADER]: Becoming Leader. State: Replica: 34598cea9ca64d5ea37ded12f60125cf, State: Running, Role: LEADER
I20260812 02:30:13.058243 21422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } }
I20260812 02:30:13.060782 21425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34598cea9ca64d5ea37ded12f60125cf. Latest consensus state: current_term: 3 leader_uuid: "34598cea9ca64d5ea37ded12f60125cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:13.061267 21425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:13.062052 21427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:30:13.067718 21427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:30:13.068884 21427 catalog_manager.cc:1259] Loaded cluster ID: f2332070953147fab8eb91c78e45377b
I20260812 02:30:13.068992 21427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:30:13.070470 21427 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:30:13.071475 21427 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Loaded TSK: 0
I20260812 02:30:13.072655 21427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:30:13.100603 20526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:30:13.108980 20526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:30:13.114428 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.113915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:46056)
0812 02:30:13.114129 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.114230 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 02:30:13.114578 20770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [term 3 FOLLOWER]: Refusing update from remote peer 34598cea9ca64d5ea37ded12f60125cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 02:30:13.115485 21425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 02:30:13.115720 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.115184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:46060)
0812 02:30:13.115444 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.115535 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260812 02:30:13.116266 20512 consensus_peers.cc:487] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf -> Peer c787dfba77e6402dadf24679cfedd84b (127.12.8.254:35939): Couldn't send request to peer c787dfba77e6402dadf24679cfedd84b. Status: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 02:30:13.119132 21430 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34598cea9ca64d5ea37ded12f60125cf. Latest consensus state: current_term: 3 leader_uuid: "34598cea9ca64d5ea37ded12f60125cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:13.119438 21430 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:30:13.121366 21425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "34598cea9ca64d5ea37ded12f60125cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:13.121789 21425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:13.122354 21431 mvcc.cc:205] Tried to move back new op lower bound from 7317511426508271616 to 7317511426291785728. Current Snapshot: MvccSnapshot[applied={T|T < 7317511426508271616}]
I20260812 02:30:13.125991 21430 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "34598cea9ca64d5ea37ded12f60125cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c787dfba77e6402dadf24679cfedd84b" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35939 } } peers { permanent_uuid: "34598cea9ca64d5ea37ded12f60125cf" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 37543 } attrs { promote: false } } peers { permanent_uuid: "ed8f0ac0a76142949e22cd16f7e51fce" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attrs { promote: false } } }
I20260812 02:30:13.126873 21430 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8f0ac0a76142949e22cd16f7e51fce [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:30:13.142189 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.131363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:45489 (local address 127.0.0.1:46106)
0812 02:30:13.132029 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.132058 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:13.132211 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:13.136715 (+ 4504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:13.136721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:13.136756 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:13.137085 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:13.137096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.138414 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.138420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:13.139966 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:13.139981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.141278 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.141285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:13.141467 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.141514 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 02:30:13.141613 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 02:30:13.141808 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540}
I20260812 02:30:13.143607 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.131571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.12.8.194:45489)
0812 02:30:13.132180 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:13.132187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:13.132210 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:30:13.132403 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:13.132408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:13.132502 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:13.137254 (+ 4752us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:13.137423 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.138246 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.140129 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.141038 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.141171 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.142783 (+ 1612us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:13.142847 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 02:30:13.143029 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":2109,"threads_started":1}
I20260812 02:30:13.151851 21057 tablet_service.cc:1467] Processing CreateTablet for tablet aa3d783b91af42b9834d6111a3eadcd7 (DEFAULT_TABLE table=test-table [id=c566cec52422488bb0cc00d3b9e0494a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:13.152595 21057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3d783b91af42b9834d6111a3eadcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:13.153236 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.131668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:37565 (local address 127.0.0.1:39846)
0812 02:30:13.142826 (+ 11158us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.142848 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:13.142957 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:13.145667 (+ 2710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:13.145672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:13.145700 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:13.145930 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:13.145941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.147520 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.147526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:13.149965 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:13.149990 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.152114 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.152120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:13.152329 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.152366 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 02:30:13.152445 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 02:30:13.152753 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11037,"mutex_wait_us":29,"thread_start_us":18903,"threads_started":1}
I20260812 02:30:13.153602 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.132780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.12.8.193:45959)
0812 02:30:13.136796 (+ 4016us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:13.136803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:13.145276 (+ 8473us) server_negotiation.cc:407] Connection header received
0812 02:30:13.146106 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:13.146113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:13.146225 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:13.146310 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:13.147249 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.148111 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.150760 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.151953 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.152092 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.152231 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:13.153086 (+ 855us) server_negotiation.cc:299] Negotiation successful
0812 02:30:13.153296 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3845,"thread_start_us":3682,"threads_started":1}
I20260812 02:30:13.153910 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.133131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:45959 (local address 127.0.0.1:37558)
0812 02:30:13.145123 (+ 11992us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.145142 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:13.145904 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:13.146585 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:13.146590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:13.146615 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:13.147048 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:13.147060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.148267 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.148270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:13.150501 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:13.150520 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.152493 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.152500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:13.152871 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.152913 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 02:30:13.153036 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 02:30:13.153234 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11882,"mutex_wait_us":75,"thread_start_us":11746,"threads_started":1}
I20260812 02:30:13.154243 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.133219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.12.8.195:37565)
0812 02:30:13.145332 (+ 12113us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:13.145350 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:13.145371 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:30:13.145448 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:13.145452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:13.145535 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:13.145634 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:13.146124 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.147314 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.150285 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.151901 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.153512 (+ 1611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.153646 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:13.153709 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 02:30:13.153889 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10140,"thread_start_us":9883,"threads_started":1}
I20260812 02:30:13.155396 21225 tablet_service.cc:1467] Processing CreateTablet for tablet aa3d783b91af42b9834d6111a3eadcd7 (DEFAULT_TABLE table=test-table [id=c566cec52422488bb0cc00d3b9e0494a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:13.155467 20891 tablet_service.cc:1467] Processing CreateTablet for tablet aa3d783b91af42b9834d6111a3eadcd7 (DEFAULT_TABLE table=test-table [id=c566cec52422488bb0cc00d3b9e0494a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:13.155992 21225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3d783b91af42b9834d6111a3eadcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:13.156014 20891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3d783b91af42b9834d6111a3eadcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:13.162294 21445 tablet_bootstrap.cc:492] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: Bootstrap starting.
I20260812 02:30:13.163807 21446 tablet_bootstrap.cc:492] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: Bootstrap starting.
I20260812 02:30:13.164644 21445 tablet_bootstrap.cc:654] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:13.166445 21446 tablet_bootstrap.cc:654] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:13.169926 21447 tablet_bootstrap.cc:492] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: Bootstrap starting.
I20260812 02:30:13.172040 21447 tablet_bootstrap.cc:654] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:13.172796 21446 tablet_bootstrap.cc:492] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: No bootstrap required, opened a new log
I20260812 02:30:13.172995 21446 ts_tablet_manager.cc:1397] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 02:30:13.172999 21445 tablet_bootstrap.cc:492] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: No bootstrap required, opened a new log
I20260812 02:30:13.173216 21445 ts_tablet_manager.cc:1397] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260812 02:30:13.174059 21446 raft_consensus.cc:348] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.174103 21445 raft_consensus.cc:348] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.174386 21445 raft_consensus.cc:374] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:13.174538 21446 raft_consensus.cc:374] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:13.174611 21445 raft_consensus.cc:729] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab3e5e9b4234cbdb36294fcbe6eac81, State: Initialized, Role: FOLLOWER
I20260812 02:30:13.174770 21446 raft_consensus.cc:729] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bcbb43058e4f45a71dcb78fc413b2d, State: Initialized, Role: FOLLOWER
I20260812 02:30:13.175035 21447 tablet_bootstrap.cc:492] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: No bootstrap required, opened a new log
I20260812 02:30:13.174966 21445 consensus_queue.cc:260] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.175266 21447 ts_tablet_manager.cc:1397] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 02:30:13.175349 21446 consensus_queue.cc:260] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.176360 21447 raft_consensus.cc:348] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.176717 21447 raft_consensus.cc:374] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:13.176864 21447 raft_consensus.cc:729] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572e2683344e422d87faad60b0bcb6d3, State: Initialized, Role: FOLLOWER
I20260812 02:30:13.177232 21447 consensus_queue.cc:260] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.182512 21445 ts_tablet_manager.cc:1428] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: Time spent starting tablet: real 0.009s user 0.003s sys 0.005s
I20260812 02:30:13.184628 21446 ts_tablet_manager.cc:1428] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260812 02:30:13.185617 21447 ts_tablet_manager.cc:1428] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: Time spent starting tablet: real 0.010s user 0.000s sys 0.008s
I20260812 02:30:13.187584 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.186161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:37083)
0812 02:30:13.186871 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.187181 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":674,"threads_started":1}
I20260812 02:30:13.192396 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.190154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:34251)
0812 02:30:13.191056 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.191182 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":760}
I20260812 02:30:13.197510 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.183632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:43939)
0812 02:30:13.196732 (+ 13100us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.197009 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12872,"threads_started":1}
I20260812 02:30:13.198796 21305 heartbeater.cc:503] Master 127.12.8.253:37543 was elected leader, sending a full tablet report...
I20260812 02:30:13.199402 20968 heartbeater.cc:503] Master 127.12.8.253:37543 was elected leader, sending a full tablet report...
I20260812 02:30:13.199831 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.193081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:47951)
0812 02:30:13.199193 (+ 6112us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.199444 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5939,"thread_start_us":5749,"threads_started":1}
I20260812 02:30:13.200464 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.200015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:33385)
0812 02:30:13.200186 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.200285 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 02:30:13.201277 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.200700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:48141)
0812 02:30:13.200993 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.201090 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260812 02:30:13.202122 21137 heartbeater.cc:503] Master 127.12.8.253:37543 was elected leader, sending a full tablet report...
I20260812 02:30:13.342613 21455 raft_consensus.cc:482] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:13.342983 21455 raft_consensus.cc:504] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.343932 21455 leader_election.cc:288] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489), 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565)
I20260812 02:30:13.344826 21246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3d783b91af42b9834d6111a3eadcd7" candidate_uuid: "572e2683344e422d87faad60b0bcb6d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" is_pre_election: true
I20260812 02:30:13.344815 21081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3d783b91af42b9834d6111a3eadcd7" candidate_uuid: "572e2683344e422d87faad60b0bcb6d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" is_pre_election: true
I20260812 02:30:13.345155 21246 raft_consensus.cc:2454] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572e2683344e422d87faad60b0bcb6d3 in term 0.
I20260812 02:30:13.345160 21081 raft_consensus.cc:2454] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572e2683344e422d87faad60b0bcb6d3 in term 0.
I20260812 02:30:13.345778 20837 leader_election.cc:302] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572e2683344e422d87faad60b0bcb6d3, 78bcbb43058e4f45a71dcb78fc413b2d; no voters:
I20260812 02:30:13.346101 21455 raft_consensus.cc:2781] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:30:13.346258 21455 raft_consensus.cc:482] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:13.346365 21455 raft_consensus.cc:3037] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:13.349053 21455 raft_consensus.cc:504] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.349860 21455 leader_election.cc:288] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [CANDIDATE]: Term 1 election: Requested vote from peers 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489), 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565)
I20260812 02:30:13.350730 21246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3d783b91af42b9834d6111a3eadcd7" candidate_uuid: "572e2683344e422d87faad60b0bcb6d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bcbb43058e4f45a71dcb78fc413b2d"
I20260812 02:30:13.350775 21081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3d783b91af42b9834d6111a3eadcd7" candidate_uuid: "572e2683344e422d87faad60b0bcb6d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81"
I20260812 02:30:13.350983 21246 raft_consensus.cc:3037] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:13.351091 21081 raft_consensus.cc:3037] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:13.353657 21246 raft_consensus.cc:2454] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572e2683344e422d87faad60b0bcb6d3 in term 1.
I20260812 02:30:13.353857 21081 raft_consensus.cc:2454] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572e2683344e422d87faad60b0bcb6d3 in term 1.
I20260812 02:30:13.354322 20837 leader_election.cc:302] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572e2683344e422d87faad60b0bcb6d3, 78bcbb43058e4f45a71dcb78fc413b2d; no voters:
I20260812 02:30:13.354717 21455 raft_consensus.cc:2781] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:13.355341 21455 raft_consensus.cc:686] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 1 LEADER]: Becoming Leader. State: Replica: 572e2683344e422d87faad60b0bcb6d3, State: Running, Role: LEADER
I20260812 02:30:13.355831 21455 consensus_queue.cc:237] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } }
I20260812 02:30:13.356860 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.356130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:53183)
0812 02:30:13.356470 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.356616 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260812 02:30:13.359498 20527 catalog_manager.cc:5495] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572e2683344e422d87faad60b0bcb6d3 (127.12.8.193). New cstate: current_term: 1 leader_uuid: "572e2683344e422d87faad60b0bcb6d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572e2683344e422d87faad60b0bcb6d3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 45959 } health_report { overall_health: HEALTHY } } }
I20260812 02:30:13.395167 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.389358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.12.8.193:45959)
0812 02:30:13.389627 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:13.389633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:13.389659 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:30:13.389794 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:13.389800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:13.389895 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:13.389984 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:13.390516 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.391294 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.392713 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.393374 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.393645 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.394322 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:13.394467 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:13.394553 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:13.394711 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 02:30:13.394815 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:30:13.399746 21081 raft_consensus.cc:1261] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 1 FOLLOWER]: Refusing update from remote peer 572e2683344e422d87faad60b0bcb6d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:13.399746 21246 raft_consensus.cc:1261] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 1 FOLLOWER]: Refusing update from remote peer 572e2683344e422d87faad60b0bcb6d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:13.400625 21459 consensus_queue.cc:1035] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bcbb43058e4f45a71dcb78fc413b2d" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:13.401105 21455 consensus_queue.cc:1035] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab3e5e9b4234cbdb36294fcbe6eac81" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:13.404564 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.403818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:48223)
0812 02:30:13.404002 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.404258 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 02:30:13.404861 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.404267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:49445)
0812 02:30:13.404515 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.404635 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260812 02:30:13.413079 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.412251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.194:36643)
0812 02:30:13.412693 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.412832 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
I20260812 02:30:13.414577 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.413921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.193:43815)
0812 02:30:13.414259 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.414367 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260812 02:30:13.416177 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.415389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.12.8.195:37391)
0812 02:30:13.415754 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.415890 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260812 02:30:13.422333 20526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:13.422716 20526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:13.430500 20526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34598cea9ca64d5ea37ded12f60125cf: Sending DeleteTablet for 3 replicas of tablet aa3d783b91af42b9834d6111a3eadcd7
I20260812 02:30:13.431595 21057 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3d783b91af42b9834d6111a3eadcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:13 UTC) from {username='oryx', principal='oryx/127.12.8.253@KRBTEST.COM'} at 127.0.0.1:46106
I20260812 02:30:13.431613 21225 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3d783b91af42b9834d6111a3eadcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:13 UTC) from {username='oryx', principal='oryx/127.12.8.253@KRBTEST.COM'} at 127.0.0.1:39846
I20260812 02:30:13.431818 21224 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3d783b91af42b9834d6111a3eadcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:13 UTC) from {username='oryx', principal='oryx/127.12.8.253@KRBTEST.COM'} at 127.0.0.1:39846
I20260812 02:30:13.432256 20891 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3d783b91af42b9834d6111a3eadcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:13 UTC) from {username='oryx', principal='oryx/127.12.8.253@KRBTEST.COM'} at 127.0.0.1:37558
I20260812 02:30:13.432729 21479 tablet_replica.cc:330] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: stopping tablet replica
I20260812 02:30:13.432971 21479 raft_consensus.cc:2229] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:13.433403 21479 raft_consensus.cc:2258] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:13.434310 21478 tablet_replica.cc:330] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: stopping tablet replica
I20260812 02:30:13.434690 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 20724
I20260812 02:30:13.434669 21478 raft_consensus.cc:2229] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:13.435029 21478 raft_consensus.cc:2258] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:13.436576 21478 ts_tablet_manager.cc:1905] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:13.439414 21479 ts_tablet_manager.cc:1905] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:13.445582 21480 tablet_replica.cc:330] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: stopping tablet replica
I20260812 02:30:13.445600 21478 ts_tablet_manager.cc:1918] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:13.445780 21478 log.cc:1196] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/aa3d783b91af42b9834d6111a3eadcd7
I20260812 02:30:13.446200 21480 raft_consensus.cc:2229] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:30:13.446357 21478 ts_tablet_manager.cc:1939] T aa3d783b91af42b9834d6111a3eadcd7 P 78bcbb43058e4f45a71dcb78fc413b2d: Deleting consensus metadata
I20260812 02:30:13.446867 21480 raft_consensus.cc:2258] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:13.447952 20510 catalog_manager.cc:4841] TS 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565): tablet aa3d783b91af42b9834d6111a3eadcd7 (table test-table [id=c566cec52422488bb0cc00d3b9e0494a]) successfully deleted
W20260812 02:30:13.448731 20510 catalog_manager.cc:4808] TS 78bcbb43058e4f45a71dcb78fc413b2d (127.12.8.195:37565): delete failed for tablet aa3d783b91af42b9834d6111a3eadcd7 because the tablet was not found. No further retry: Not found: Tablet not found: aa3d783b91af42b9834d6111a3eadcd7
I20260812 02:30:13.448771 21479 ts_tablet_manager.cc:1918] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:13.448848 21480 ts_tablet_manager.cc:1905] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:13.448933 21479 log.cc:1196] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/aa3d783b91af42b9834d6111a3eadcd7
I20260812 02:30:13.449481 21479 ts_tablet_manager.cc:1939] T aa3d783b91af42b9834d6111a3eadcd7 P 0ab3e5e9b4234cbdb36294fcbe6eac81: Deleting consensus metadata
I20260812 02:30:13.450862 20510 catalog_manager.cc:4841] TS 0ab3e5e9b4234cbdb36294fcbe6eac81 (127.12.8.194:45489): tablet aa3d783b91af42b9834d6111a3eadcd7 (table test-table [id=c566cec52422488bb0cc00d3b9e0494a]) successfully deleted
I20260812 02:30:13.456019 21480 ts_tablet_manager.cc:1918] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:13.456215 21480 log.cc:1196] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/aa3d783b91af42b9834d6111a3eadcd7
I20260812 02:30:13.456732 21480 ts_tablet_manager.cc:1939] T aa3d783b91af42b9834d6111a3eadcd7 P 572e2683344e422d87faad60b0bcb6d3: Deleting consensus metadata
I20260812 02:30:13.458290 20512 catalog_manager.cc:4841] TS 572e2683344e422d87faad60b0bcb6d3 (127.12.8.193:45959): tablet aa3d783b91af42b9834d6111a3eadcd7 (table test-table [id=c566cec52422488bb0cc00d3b9e0494a]) successfully deleted
I20260812 02:30:13.466672 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.435783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.12.8.193:45959)
0812 02:30:13.435996 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:13.436003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:13.447803 (+ 11800us) server_negotiation.cc:407] Connection header received
0812 02:30:13.449629 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:13.449636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:13.449760 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:13.449917 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:13.451840 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.452722 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.454695 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.455270 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.455406 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.459159 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:13.459207 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:13.459229 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:13.459279 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:13.463114 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:13.464031 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:13.464040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:13.464043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:13.464241 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:13.464761 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:13.464766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:13.464773 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:13.465370 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:13.465654 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:13.466063 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 02:30:13.466329 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:30:13.467552 20958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:13.610987 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 20724
I20260812 02:30:13.646842 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.646259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:46074)
0812 02:30:13.646509 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.646631 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 02:30:13.646852 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 20980
I20260812 02:30:13.648428 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.647980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:35939 (local address 127.0.0.1:46090)
0812 02:30:13.648153 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:13.648254 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 02:30:13.663478 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.647836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.12.8.194:45489)
0812 02:30:13.648776 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:13.648781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:13.648798 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:30:13.648863 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:13.648868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:13.648944 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:13.649056 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:13.650531 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.651316 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.652714 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.653237 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.653363 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.656707 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:13.656728 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:13.656750 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:13.656797 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:13.660152 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:13.660987 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:13.660995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:13.660998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:13.661435 (+ 437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:13.661816 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:13.661820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:13.661822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:13.662283 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:13.662562 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:13.662925 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 02:30:13.663140 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":705,"threads_started":1}
I20260812 02:30:13.664286 21130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:13.770280 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 20980
I20260812 02:30:13.800549 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 21149
I20260812 02:30:13.817772 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:13.801685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.12.8.195:37565)
0812 02:30:13.802234 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:13.802239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:13.802257 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:30:13.802330 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:13.802334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:13.802411 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:13.802516 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:13.804152 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.804955 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.806445 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:13.806906 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:13.807054 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:13.810220 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:13.810244 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:13.810266 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:13.810318 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:13.814292 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:13.815258 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:13.815267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:13.815270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:13.815522 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:13.815948 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:13.815952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:13.815954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:13.816459 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:13.816757 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:13.817128 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 02:30:13.817418 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
I20260812 02:30:13.818616 21297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:13.939107 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 21149
I20260812 02:30:13.968755 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 20370
I20260812 02:30:13.970337 20562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:14.113435 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 20370
I20260812 02:30:14.146996 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 20488
I20260812 02:30:14.148409 20793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:14.273633 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 20488
2026-08-12T02:30:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (13349 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:30:14 dist-test-slave-vq2q krb5kdc[21500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:30:14 dist-test-slave-vq2q krb5kdc[21500](info): set up 2 sockets
Aug 12 02:30:14 dist-test-slave-vq2q krb5kdc[21500](info): commencing operation
krb5kdc: starting...
W20260812 02:30:15.452026 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:30:15 dist-test-slave-vq2q krb5kdc[21500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:30:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.254.keytab.
Entry for principal oryx/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.254.keytab.
I20260812 02:30:15.811798 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45337
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.254.keytab
--principal=oryx/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:16.232749 21516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:16.233115 21516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:16.233249 21516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:16.240818 21516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:16.240952 21516 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:30:16.240995 21516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:16.241050 21516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:16.241135 21516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:16.241250 21516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:16.251873 21516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:45337
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.254.keytab
--principal=oryx/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:16.253427 21516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:16.254842 21516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:16.268529 21525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:16.268529 21529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:16.268600 21527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:16.270282 21516 server_base.cc:1034] running on GCE node
I20260812 02:30:16.271250 21516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:16.272955 21516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:16.274214 21516 hybrid_clock.cc:648] HybridClock initialized: now 1786501816274124 us; error 74 us; skew 500 ppm
I20260812 02:30:16.274588 21516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:16 dist-test-slave-vq2q krb5kdc[21500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501816, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:16.278185 21516 init.cc:377] Logged in from keytab as oryx/127.12.8.254@KRBTEST.COM (short username oryx)
I20260812 02:30:16.280026 21516 webserver.cc:460] Webserver started at http://127.12.8.254:33611/ using document root <none> and password file <none>
I20260812 02:30:16.280812 21516 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:16.280928 21516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:16.281339 21516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:16.284562 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "d0a8bdde77b44e368ad78f90272480b2"
format_stamp: "Formatted at 2026-08-12 02:30:16 on dist-test-slave-vq2q"
server_key: "828bcb268cb9365cf552b2b318b87ef4"
server_key_iv: "f530245ac95e44efe9e8c77b25bfd81f"
server_key_version: "encryptionkey@0"
I20260812 02:30:16.285458 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "d0a8bdde77b44e368ad78f90272480b2"
format_stamp: "Formatted at 2026-08-12 02:30:16 on dist-test-slave-vq2q"
server_key: "828bcb268cb9365cf552b2b318b87ef4"
server_key_iv: "f530245ac95e44efe9e8c77b25bfd81f"
server_key_version: "encryptionkey@0"
I20260812 02:30:16.291522 21516 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:30:16.295980 21536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:16.297885 21516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:30:16.298134 21516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "d0a8bdde77b44e368ad78f90272480b2"
format_stamp: "Formatted at 2026-08-12 02:30:16 on dist-test-slave-vq2q"
server_key: "828bcb268cb9365cf552b2b318b87ef4"
server_key_iv: "f530245ac95e44efe9e8c77b25bfd81f"
server_key_version: "encryptionkey@0"
I20260812 02:30:16.298372 21516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:16.326478 21516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:16.331343 21516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:16.331617 21516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:16.356228 21516 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:45337
I20260812 02:30:16.356212 21610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:45337 every 8 connection(s)
I20260812 02:30:16.357774 21516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:30:16.358719 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 21516
I20260812 02:30:16.358971 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:30:16.359822 12323 external_mini_cluster.cc:1428] Setting key a8a1e10ca6931c76df789899329254de
I20260812 02:30:16.362808 21611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:16.377041 21611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: Bootstrap starting.
Aug 12 02:30:16 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:16.381171 21611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:16.382659 21611 log.cc:824] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:16.386727 21611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: No bootstrap required, opened a new log
I20260812 02:30:16.388573 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:16.362443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.12.8.254:45337)
0812 02:30:16.363414 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:16.363430 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:16.363487 (+ 57us) server_negotiation.cc:407] Connection header received
0812 02:30:16.365567 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:16.365599 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:16.366182 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:16.366668 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:16.368675 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:16.370410 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:16.372264 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:16.373167 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:16.373415 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:16.378836 (+ 5421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:16.378850 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:16.378871 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:16.378953 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:16.383124 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:16.384337 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:16.384344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:16.384351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:16.384571 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:16.385023 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:16.385028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:16.385031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:16.385828 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:16.386149 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:16.386661 (+ 512us) server_negotiation.cc:299] Negotiation successful
0812 02:30:16.387142 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":421,"threads_started":1}
I20260812 02:30:16.396112 21611 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a8bdde77b44e368ad78f90272480b2" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45337 } }
I20260812 02:30:16.396543 21611 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:16.396627 21611 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a8bdde77b44e368ad78f90272480b2, State: Initialized, Role: FOLLOWER
I20260812 02:30:16.397365 21611 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a8bdde77b44e368ad78f90272480b2" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45337 } }
I20260812 02:30:16.397620 21611 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:30:16.397789 21611 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:30:16.397956 21611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:16.400251 21611 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a8bdde77b44e368ad78f90272480b2" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45337 } }
I20260812 02:30:16.400830 21611 leader_election.cc:302] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a8bdde77b44e368ad78f90272480b2; no voters:
I20260812 02:30:16.401647 21611 leader_election.cc:288] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:30:16.401845 21618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:16.402820 21618 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [term 1 LEADER]: Becoming Leader. State: Replica: d0a8bdde77b44e368ad78f90272480b2, State: Running, Role: LEADER
I20260812 02:30:16.403645 21618 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a8bdde77b44e368ad78f90272480b2" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45337 } }
I20260812 02:30:16.404337 21611 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:16.407420 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a8bdde77b44e368ad78f90272480b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a8bdde77b44e368ad78f90272480b2" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45337 } } }
I20260812 02:30:16.407387 21622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a8bdde77b44e368ad78f90272480b2. Latest consensus state: current_term: 1 leader_uuid: "d0a8bdde77b44e368ad78f90272480b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a8bdde77b44e368ad78f90272480b2" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45337 } } }
I20260812 02:30:16.407979 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:16.407989 21622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:16.411391 21627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:30:16.419049 21627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:30:16.434850 21627 catalog_manager.cc:1347] Generated new cluster ID: a3f13b66bce64fda982deea51edbe7f4
I20260812 02:30:16.435011 21627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:30:16.456192 21627 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:30:16.458001 21627 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:30:16.468425 21627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: Generated new TSK 0
I20260812 02:30:16.469707 21627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab.
Entry for principal oryx/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab.
I20260812 02:30:16.557832 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45337
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab
--principal=oryx/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:16.994005 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:16.994410 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:16.994586 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:17.003978 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:17.004276 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:17.004469 21648 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:30:17.004637 21648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:30:17.019187 21648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab
--principal=oryx/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:17.021021 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:17.022851 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:17.041219 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:17.041612 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:17.042440 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:17.042538 21648 server_base.cc:1034] running on GCE node
I20260812 02:30:17.043531 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:17.044762 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:17.046027 21648 hybrid_clock.cc:648] HybridClock initialized: now 1786501817045952 us; error 48 us; skew 500 ppm
I20260812 02:30:17.046361 21648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:17 dist-test-slave-vq2q krb5kdc[21500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501817, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:17.050261 21648 init.cc:377] Logged in from keytab as oryx/127.12.8.193@KRBTEST.COM (short username oryx)
I20260812 02:30:17.052243 21648 webserver.cc:460] Webserver started at http://127.12.8.193:43049/ using document root <none> and password file <none>
I20260812 02:30:17.053112 21648 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:17.053262 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:17.053650 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:17.057318 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "22a46d39421647f991617549a491783c"
format_stamp: "Formatted at 2026-08-12 02:30:17 on dist-test-slave-vq2q"
server_key: "ff12839688d5628b53a556403926af8c"
server_key_iv: "8109de639b791d6f0e361559ca5f8cb5"
server_key_version: "encryptionkey@0"
I20260812 02:30:17.058212 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "22a46d39421647f991617549a491783c"
format_stamp: "Formatted at 2026-08-12 02:30:17 on dist-test-slave-vq2q"
server_key: "ff12839688d5628b53a556403926af8c"
server_key_iv: "8109de639b791d6f0e361559ca5f8cb5"
server_key_version: "encryptionkey@0"
I20260812 02:30:17.064713 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 02:30:17.069478 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:17.071386 21648 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:30:17.071587 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "22a46d39421647f991617549a491783c"
format_stamp: "Formatted at 2026-08-12 02:30:17 on dist-test-slave-vq2q"
server_key: "ff12839688d5628b53a556403926af8c"
server_key_iv: "8109de639b791d6f0e361559ca5f8cb5"
server_key_version: "encryptionkey@0"
I20260812 02:30:17.071842 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:17.106976 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:17.111631 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:17.111908 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:17.115597 21648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:17.115692 21648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:17.115849 21648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:17.115912 21648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:30:17 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501817, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:17.159829 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:17.117607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45337 (local address 127.12.8.193:42657)
0812 02:30:17.131321 (+ 13714us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:17.131399 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:17.134119 (+ 2720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:17.135434 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:17.135448 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:17.136387 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:17.137608 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:17.137643 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.139324 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.139333 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:17.141937 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:17.141959 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.143496 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.143509 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:17.143832 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:17.144732 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:17.144765 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:17.148482 (+ 3717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:17.154167 (+ 5685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:17.154178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:17.154200 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:17.154929 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:17.155449 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:17.155455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:17.155459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:17.155795 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:17.156532 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:17.156543 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:17.157079 (+ 536us) client_negotiation.cc:769] Sending connection context
0812 02:30:17.157506 (+ 427us) client_negotiation.cc:240] Negotiation successful
0812 02:30:17.158154 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13255,"thread_start_us":17192,"threads_started":1}
I20260812 02:30:17.161319 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:17.117023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:42657 (local address 127.12.8.254:45337)
0812 02:30:17.133547 (+ 16524us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:17.133554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:17.133579 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:30:17.134623 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:17.134632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:17.134742 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:17.134890 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:17.137876 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.139060 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.142179 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.142878 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.143014 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:17.148831 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:17.148839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:17.148845 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:17.148895 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:17.153952 (+ 5057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:17.155099 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:17.155106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:17.155110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:17.155320 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:17.155946 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:17.155950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:17.155954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:17.156381 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:17.156547 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:17.160619 (+ 4072us) server_negotiation.cc:299] Negotiation successful
0812 02:30:17.160896 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16391,"thread_start_us":16288,"threads_started":1}
I20260812 02:30:17.191978 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:17.172954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45337 (local address 127.12.8.193:37365)
0812 02:30:17.174737 (+ 1783us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:17.174759 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:17.175003 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:17.175651 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:17.175656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:17.176458 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:17.177105 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:17.177122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.178765 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.178771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:17.180694 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:17.180716 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.182184 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.182192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:17.182431 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:17.183254 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:17.183284 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:17.184858 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:17.188837 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:17.188845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:17.188849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:17.189336 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:17.189760 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:17.189765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:17.189769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:17.189915 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:17.190579 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:17.190583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:17.190827 (+ 244us) client_negotiation.cc:769] Sending connection context
0812 02:30:17.190991 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 02:30:17.191431 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1607,"thread_start_us":1467,"threads_started":1}
I20260812 02:30:17.193065 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:17.173271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:37365 (local address 127.12.8.254:45337)
0812 02:30:17.173495 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:17.173502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:17.174862 (+ 1360us) server_negotiation.cc:407] Connection header received
0812 02:30:17.175176 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:17.175182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:17.175296 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:17.175368 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:17.177338 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.178574 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.180876 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.181769 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.181919 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:17.185121 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:17.185127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:17.185133 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:17.185198 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:17.188649 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:17.189461 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:17.189467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:17.189471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:17.189630 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:17.190039 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:17.190044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:17.190048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:17.190421 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:17.192303 (+ 1882us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:17.192464 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 02:30:17.192720 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 02:30:17.200641 21648 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:43315
I20260812 02:30:17.200642 21803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:43315 every 8 connection(s)
I20260812 02:30:17.202402 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:30:17.205119 21804 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45337
I20260812 02:30:17.205374 21804 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:17.206178 21804 heartbeater.cc:511] Master 127.12.8.254:45337 requested a full tablet report, sending...
I20260812 02:30:17.208564 21557 ts_manager.cc:194] Registered new tserver with Master: 22a46d39421647f991617549a491783c (127.12.8.193:43315)
I20260812 02:30:17.208760 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 21648
I20260812 02:30:17.208961 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:30:17.209750 12323 external_mini_cluster.cc:1428] Setting key d538a9bca2ff48a1798f7c6a130c85a6
I20260812 02:30:17.210848 21557 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:42657
WARNING: no policy specified for oryx/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab.
Entry for principal oryx/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab.
I20260812 02:30:17.272425 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45337
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab
--principal=oryx/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:17.638162 21808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:17.638480 21808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:17.638613 21808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:17.646163 21808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:17.646289 21808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:17.646413 21808 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:30:17.646533 21808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:30:17.656867 21808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab
--principal=oryx/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:17.658278 21808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:17.659759 21808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:17.673240 21814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:17.673273 21816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:17.673378 21813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:17.674086 21808 server_base.cc:1034] running on GCE node
I20260812 02:30:17.674682 21808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:17.675714 21808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:17.676965 21808 hybrid_clock.cc:648] HybridClock initialized: now 1786501817676877 us; error 67 us; skew 500 ppm
I20260812 02:30:17.677275 21808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:17 dist-test-slave-vq2q krb5kdc[21500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501817, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:17.680423 21808 init.cc:377] Logged in from keytab as oryx/127.12.8.194@KRBTEST.COM (short username oryx)
I20260812 02:30:17.681970 21808 webserver.cc:460] Webserver started at http://127.12.8.194:42979/ using document root <none> and password file <none>
I20260812 02:30:17.682650 21808 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:17.682754 21808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:17.683104 21808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:17.686116 21808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "c2310e5e04cc47ccbeb16ed9efd51421"
format_stamp: "Formatted at 2026-08-12 02:30:17 on dist-test-slave-vq2q"
server_key: "fb61a525df64da9ca7a88306f7593217"
server_key_iv: "86c6230122f411a001635d009a0ec74e"
server_key_version: "encryptionkey@0"
I20260812 02:30:17.686928 21808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "c2310e5e04cc47ccbeb16ed9efd51421"
format_stamp: "Formatted at 2026-08-12 02:30:17 on dist-test-slave-vq2q"
server_key: "fb61a525df64da9ca7a88306f7593217"
server_key_iv: "86c6230122f411a001635d009a0ec74e"
server_key_version: "encryptionkey@0"
I20260812 02:30:17.692312 21808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:30:17.696143 21825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:17.697702 21808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:30:17.697881 21808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "c2310e5e04cc47ccbeb16ed9efd51421"
format_stamp: "Formatted at 2026-08-12 02:30:17 on dist-test-slave-vq2q"
server_key: "fb61a525df64da9ca7a88306f7593217"
server_key_iv: "86c6230122f411a001635d009a0ec74e"
server_key_version: "encryptionkey@0"
I20260812 02:30:17.698093 21808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:17.735307 21808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:17.740021 21808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:17.740286 21808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:17.743248 21808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:17.743424 21808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:17.743597 21808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:17.743674 21808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:30:17 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501817, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:17.768540 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:17.744116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:41021 (local address 127.12.8.254:45337)
0812 02:30:17.744617 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:17.744622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:17.745879 (+ 1257us) server_negotiation.cc:407] Connection header received
0812 02:30:17.747950 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:17.747957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:17.748039 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:17.748136 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:17.750424 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.751644 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.754227 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.754903 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.755044 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:17.760483 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:17.760491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:17.760496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:17.760548 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:17.764166 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:17.765536 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:17.765543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:17.765547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:17.765758 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:17.766388 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:17.766391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:17.766394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:17.766690 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:17.766820 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:17.767996 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0812 02:30:17.768214 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":287,"threads_started":1}
I20260812 02:30:17.770576 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:17.744822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45337 (local address 127.12.8.194:41021)
0812 02:30:17.745682 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:17.745748 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:17.747660 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:17.748448 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:17.748461 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:17.749372 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:17.750198 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:17.750225 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.751860 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.751866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:17.754051 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:17.754070 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.755212 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.755222 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:17.755464 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:17.756392 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:17.756438 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:17.760140 (+ 3702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:17.764435 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:17.764449 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:17.764471 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:17.765380 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:17.765892 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:17.765897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:17.765902 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:17.766242 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:17.766810 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:17.766821 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:17.767445 (+ 624us) client_negotiation.cc:769] Sending connection context
0812 02:30:17.767846 (+ 401us) client_negotiation.cc:240] Negotiation successful
0812 02:30:17.768526 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":377,"threads_started":1}
I20260812 02:30:17.799059 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:17.781909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45337 (local address 127.12.8.194:60681)
0812 02:30:17.782902 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:17.782926 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:17.783194 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:17.783801 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:17.783805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:17.784528 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:17.784958 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:17.784972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.786400 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.786405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:17.788361 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:17.788384 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.789771 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.789779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:17.789963 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:17.790686 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:17.790717 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:17.792070 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:17.795925 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:17.795931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:17.795936 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:17.796605 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:17.797020 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:17.797025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:17.797027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:17.797169 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:17.797846 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:17.797850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:17.798055 (+ 205us) client_negotiation.cc:769] Sending connection context
0812 02:30:17.798230 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 02:30:17.798638 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":598,"threads_started":1}
I20260812 02:30:17.799126 21808 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:34963
I20260812 02:30:17.799166 21972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:34963 every 8 connection(s)
I20260812 02:30:17.799975 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:17.781768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:60681 (local address 127.12.8.254:45337)
0812 02:30:17.781993 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:17.782000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:17.783037 (+ 1037us) server_negotiation.cc:407] Connection header received
0812 02:30:17.783393 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:17.783400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:17.783496 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:17.783570 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:17.785160 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.786203 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.788612 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:17.789291 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:17.789437 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:17.792391 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:17.792399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:17.792404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:17.792448 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:17.795764 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:17.796737 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:17.796743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:17.796746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:17.796883 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:17.797325 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:17.797329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:17.797333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:17.797703 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:17.797856 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:17.799559 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0812 02:30:17.799761 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 02:30:17.800580 21808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:30:17.802974 21973 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45337
I20260812 02:30:17.803220 21973 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:17.804105 21973 heartbeater.cc:511] Master 127.12.8.254:45337 requested a full tablet report, sending...
I20260812 02:30:17.805071 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 21808
I20260812 02:30:17.805245 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:30:17.805557 21557 ts_manager.cc:194] Registered new tserver with Master: c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963)
I20260812 02:30:17.805733 12323 external_mini_cluster.cc:1428] Setting key d14b8f0ff54ef0b68d82a92cdd73183d
I20260812 02:30:17.806488 21557 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:41021
WARNING: no policy specified for oryx/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab.
Entry for principal oryx/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab.
I20260812 02:30:17.870810 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45337
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab
--principal=oryx/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:30:18.214730 21804 heartbeater.cc:503] Master 127.12.8.254:45337 was elected leader, sending a full tablet report...
W20260812 02:30:18.216759 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:18.217136 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:18.217300 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:18.224732 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:18.224857 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:18.224970 21979 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:30:18.225090 21979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:18.235582 21979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab
--principal=oryx/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.254:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:18.237025 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:18.238526 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:18.253212 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:18.253298 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:18.253880 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:18.254123 21979 server_base.cc:1034] running on GCE node
I20260812 02:30:18.254844 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:18.255920 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:18.257205 21979 hybrid_clock.cc:648] HybridClock initialized: now 1786501818257101 us; error 62 us; skew 500 ppm
I20260812 02:30:18.257562 21979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:18 dist-test-slave-vq2q krb5kdc[21500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501818, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:18.266338 21979 init.cc:377] Logged in from keytab as oryx/127.12.8.195@KRBTEST.COM (short username oryx)
I20260812 02:30:18.268180 21979 webserver.cc:460] Webserver started at http://127.12.8.195:35559/ using document root <none> and password file <none>
I20260812 02:30:18.269029 21979 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:18.269132 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:18.269490 21979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:18.272665 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "422605f8283c4081859176fc8065e3b1"
format_stamp: "Formatted at 2026-08-12 02:30:18 on dist-test-slave-vq2q"
server_key: "9d244301a5c7e5712e8b058a14ab0c4c"
server_key_iv: "eab0bdd014b858c3bf1b309e46835c6f"
server_key_version: "encryptionkey@0"
I20260812 02:30:18.273576 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "422605f8283c4081859176fc8065e3b1"
format_stamp: "Formatted at 2026-08-12 02:30:18 on dist-test-slave-vq2q"
server_key: "9d244301a5c7e5712e8b058a14ab0c4c"
server_key_iv: "eab0bdd014b858c3bf1b309e46835c6f"
server_key_version: "encryptionkey@0"
I20260812 02:30:18.279387 21979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:30:18.283631 21998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:18.285301 21979 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:30:18.285468 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "422605f8283c4081859176fc8065e3b1"
format_stamp: "Formatted at 2026-08-12 02:30:18 on dist-test-slave-vq2q"
server_key: "9d244301a5c7e5712e8b058a14ab0c4c"
server_key_iv: "eab0bdd014b858c3bf1b309e46835c6f"
server_key_version: "encryptionkey@0"
I20260812 02:30:18.285681 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:18.307853 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:18.312263 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:18.312517 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:18.315832 21979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:18.315924 21979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:18.316082 21979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:18.316159 21979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:30:18 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501818, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:18.347834 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.316971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:45325 (local address 127.12.8.254:45337)
0812 02:30:18.317934 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.317942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.319428 (+ 1486us) server_negotiation.cc:407] Connection header received
0812 02:30:18.322370 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.322379 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.322499 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.322677 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:18.326326 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.327571 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.330432 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.331181 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.331347 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.337349 (+ 6002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:18.337356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:18.337360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:18.337411 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:18.341910 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:18.344011 (+ 2101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:18.344018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:18.344022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:18.344262 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:18.345224 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:18.345230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:18.345233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:18.345692 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:18.345891 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.347002 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.347338 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":605,"threads_started":1}
I20260812 02:30:18.349073 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.318098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45337 (local address 127.12.8.195:45325)
0812 02:30:18.319194 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:18.319257 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:18.321950 (+ 2693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:18.323103 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:18.323117 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:18.324273 (+ 1156us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:18.325804 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.325855 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.327874 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.327879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.330213 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.330236 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.331521 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.331528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.331786 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.332630 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:18.332665 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:18.336947 (+ 4282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:18.342773 (+ 5826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:18.342786 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:18.342811 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:18.343740 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:18.344485 (+ 745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:18.344496 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:18.344501 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:18.344970 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:18.345918 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:18.345930 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:18.346488 (+ 558us) client_negotiation.cc:769] Sending connection context
0812 02:30:18.346836 (+ 348us) client_negotiation.cc:240] Negotiation successful
0812 02:30:18.347435 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":443,"threads_started":1}
I20260812 02:30:18.372793 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.356958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45337 (local address 127.12.8.195:55037)
0812 02:30:18.357948 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:18.357969 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:18.358150 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:18.358527 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:18.358532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:18.359131 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:18.359502 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.359512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.360765 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.360769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.362010 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.362025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.362884 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.362890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.363070 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.363867 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:18.363897 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:18.364943 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:18.368910 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:18.368917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:18.368922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:18.369526 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:18.370110 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:18.370118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:18.370122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:18.370422 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:18.371360 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:18.371366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:18.371794 (+ 428us) client_negotiation.cc:769] Sending connection context
0812 02:30:18.371981 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 02:30:18.372376 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":566,"threads_started":1}
I20260812 02:30:18.373054 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.356940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:55037 (local address 127.12.8.254:45337)
0812 02:30:18.357167 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.357229 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.358080 (+ 851us) server_negotiation.cc:407] Connection header received
0812 02:30:18.358293 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.358300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.358379 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.358468 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:18.359672 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.360572 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.362181 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.362658 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.362781 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.365233 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:18.365241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:18.365246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:18.365301 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:18.368716 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:18.369682 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:18.369688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:18.369692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:18.369918 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:18.370639 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:18.370645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:18.370649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:18.371145 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:18.371719 (+ 574us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.372208 (+ 489us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.372598 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 02:30:18.387388 21979 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:46785
I20260812 02:30:18.387432 22152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:46785 every 8 connection(s)
I20260812 02:30:18.388980 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:18.391722 22153 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45337
I20260812 02:30:18.391745 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 21979
I20260812 02:30:18.391917 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:30:18.392009 22153 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:18.392472 12323 external_mini_cluster.cc:1428] Setting key b70e692b8fedcf5b04a12fa03e812666
I20260812 02:30:18.392769 22153 heartbeater.cc:511] Master 127.12.8.254:45337 requested a full tablet report, sending...
I20260812 02:30:18.394745 21556 ts_manager.cc:194] Registered new tserver with Master: 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:18.396159 21556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:45325
I20260812 02:30:18.396570 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:30:18 dist-test-slave-vq2q krb5kdc[21500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501818, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:30:18 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:18 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:30:18.472379 22010 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:18.459832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.12.8.254:45337)
0812 02:30:18.460118 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.460125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.460269 (+ 144us) server_negotiation.cc:407] Connection header received
0812 02:30:18.460489 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.460495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.460599 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.460691 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:18.462207 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.462962 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.464297 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.464721 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.464842 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.472000 (+ 7158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56178: BlockingRecv error: recv got EOF from 127.0.0.1:56178 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 12 02:30:18 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501818, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:18.492317 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.476882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.12.8.254:45337)
0812 02:30:18.477087 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.477092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.477396 (+ 304us) server_negotiation.cc:407] Connection header received
0812 02:30:18.477583 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.477588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.477667 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.477761 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:18.479043 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.479824 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.481124 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.481664 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.481789 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.485689 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:18.485695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:18.485699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:18.485761 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:18.489272 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:18.490147 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:18.490153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:18.490159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:18.490371 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:18.490804 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:18.490808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:18.490810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:18.491215 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:18.491462 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.491734 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.491986 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:30:18.497967 21556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:30:18.506901 21556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:30:18.543216 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.524079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:43315 (local address 127.0.0.1:47736)
0812 02:30:18.533713 (+ 9634us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:18.533741 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:18.533925 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:18.535207 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:18.535215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:18.535253 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:18.535665 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.535678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.537490 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.537496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.539656 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.539671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.542103 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.542109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.542334 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.542378 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 02:30:18.542521 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 02:30:18.542777 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9452,"mutex_wait_us":1,"thread_start_us":8778,"threads_started":1}
I20260812 02:30:18.543193 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.524759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.12.8.193:43315)
0812 02:30:18.534308 (+ 9549us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.534321 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.534362 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:30:18.534496 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.534506 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.534826 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.535119 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:18.535880 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.537328 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.539893 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.541514 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.541682 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.541946 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.542610 (+ 664us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.542820 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9361,"thread_start_us":9200,"threads_started":1}
I20260812 02:30:18.544384 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.522043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:34963 (local address 127.0.0.1:51604)
0812 02:30:18.523003 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:18.523048 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:18.523260 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:18.535827 (+ 12567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:18.535833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:18.535863 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:18.536439 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.536454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.538501 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.538505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.541603 (+ 3098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.541619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.543539 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.543546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.543706 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.543744 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 02:30:18.543820 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:30:18.543964 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":150,"thread_start_us":854,"threads_started":1}
I20260812 02:30:18.545495 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.522341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.12.8.194:34963)
0812 02:30:18.534535 (+ 12194us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.534546 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.534586 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:30:18.534730 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.534742 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.535521 (+ 779us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.536700 (+ 1179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:18.536959 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.538373 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.541854 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.543282 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.544684 (+ 1402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.544904 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.544981 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.545152 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11996,"thread_start_us":11858,"threads_started":1}
I20260812 02:30:18.545740 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.522586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:46785 (local address 127.0.0.1:60590)
0812 02:30:18.533279 (+ 10693us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:18.533316 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:18.533521 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:18.536184 (+ 2663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:18.536194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:18.536223 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:18.536732 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.536744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.538476 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.538483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.541274 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.541291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.544876 (+ 3585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.544884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.545023 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.545059 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 02:30:18.545147 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:30:18.545331 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10480,"mutex_wait_us":71,"thread_start_us":11039,"threads_started":1}
I20260812 02:30:18.546223 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.523293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.12.8.195:46785)
0812 02:30:18.535201 (+ 11908us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.535209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.535242 (+ 33us) server_negotiation.cc:407] Connection header received
0812 02:30:18.535343 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.535349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.535569 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.535790 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:18.537016 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.538303 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.541477 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.542992 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.543120 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.544843 (+ 1723us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.545260 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.545479 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11680,"thread_start_us":11483,"threads_started":1}
I20260812 02:30:18.547678 21722 tablet_service.cc:1467] Processing CreateTablet for tablet 3453267088384e0db23c85a09935762a (DEFAULT_TABLE table=test-table [id=672d5056339e4a64bca8deebe1520ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:18.549172 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3453267088384e0db23c85a09935762a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:18.549497 21889 tablet_service.cc:1467] Processing CreateTablet for tablet 3453267088384e0db23c85a09935762a (DEFAULT_TABLE table=test-table [id=672d5056339e4a64bca8deebe1520ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:18.551084 21889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3453267088384e0db23c85a09935762a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:18.553120 22064 tablet_service.cc:1467] Processing CreateTablet for tablet 3453267088384e0db23c85a09935762a (DEFAULT_TABLE table=test-table [id=672d5056339e4a64bca8deebe1520ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:18.554641 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3453267088384e0db23c85a09935762a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:18.566794 22179 tablet_bootstrap.cc:492] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Bootstrap starting.
I20260812 02:30:18.568770 22183 tablet_bootstrap.cc:492] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Bootstrap starting.
I20260812 02:30:18.570814 22182 tablet_bootstrap.cc:492] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Bootstrap starting.
I20260812 02:30:18.572009 22183 tablet_bootstrap.cc:654] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:18.572420 22179 tablet_bootstrap.cc:654] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:18.573909 22183 log.cc:824] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:18.574540 22179 log.cc:824] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:18.575654 22182 tablet_bootstrap.cc:654] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:18.577128 22183 tablet_bootstrap.cc:492] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: No bootstrap required, opened a new log
I20260812 02:30:18.577296 22182 log.cc:824] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:18.577489 22183 ts_tablet_manager.cc:1397] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260812 02:30:18.580822 22182 tablet_bootstrap.cc:492] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: No bootstrap required, opened a new log
I20260812 02:30:18.581149 22182 ts_tablet_manager.cc:1397] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260812 02:30:18.581387 22179 tablet_bootstrap.cc:492] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: No bootstrap required, opened a new log
I20260812 02:30:18.581697 22179 ts_tablet_manager.cc:1397] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260812 02:30:18.587967 22183 raft_consensus.cc:348] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.588498 22183 raft_consensus.cc:374] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:18.588606 22183 raft_consensus.cc:729] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422605f8283c4081859176fc8065e3b1, State: Initialized, Role: FOLLOWER
I20260812 02:30:18.589337 22183 consensus_queue.cc:260] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.591220 22153 heartbeater.cc:503] Master 127.12.8.254:45337 was elected leader, sending a full tablet report...
I20260812 02:30:18.591204 22182 raft_consensus.cc:348] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.591771 22182 raft_consensus.cc:374] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:18.591897 22182 raft_consensus.cc:729] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2310e5e04cc47ccbeb16ed9efd51421, State: Initialized, Role: FOLLOWER
I20260812 02:30:18.592089 22179 raft_consensus.cc:348] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.592540 22179 raft_consensus.cc:374] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:18.592649 22179 raft_consensus.cc:729] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a46d39421647f991617549a491783c, State: Initialized, Role: FOLLOWER
I20260812 02:30:18.592801 22182 consensus_queue.cc:260] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.593242 22183 ts_tablet_manager.cc:1428] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260812 02:30:18.593437 22179 consensus_queue.cc:260] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.594533 21973 heartbeater.cc:503] Master 127.12.8.254:45337 was elected leader, sending a full tablet report...
I20260812 02:30:18.595778 22179 ts_tablet_manager.cc:1428] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260812 02:30:18.597110 22182 ts_tablet_manager.cc:1428] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
W20260812 02:30:18.640990 22154 tablet.cc:2369] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:30:18.705595 21805 tablet.cc:2369] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:30:18.718524 22190 raft_consensus.cc:482] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:18.718999 22190 raft_consensus.cc:504] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.721392 22190 leader_election.cc:288] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a46d39421647f991617549a491783c (127.12.8.193:43315), 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:18.732050 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.721894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:44883 (local address 127.12.8.193:43315)
0812 02:30:18.722151 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.722158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.723118 (+ 960us) server_negotiation.cc:407] Connection header received
0812 02:30:18.723344 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.723350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.723443 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.723554 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:18.724576 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.725772 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.729322 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.730589 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.730830 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.730905 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.731591 (+ 686us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.731741 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:30:18.732050 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.721934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:40267 (local address 127.12.8.195:46785)
0812 02:30:18.722151 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.722158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.723114 (+ 956us) server_negotiation.cc:407] Connection header received
0812 02:30:18.723344 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.723350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.723443 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.723555 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:18.724569 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.725773 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.729325 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.730589 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.730782 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.730866 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.731591 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.731741 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:30:18.732393 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.721683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:43315 (local address 127.12.8.194:44883)
0812 02:30:18.722918 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:18.722934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:18.723137 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:18.723612 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:18.723617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:18.723665 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:18.724339 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.724361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.725970 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.725976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.729099 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.729109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.730968 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.730975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.731312 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.731397 (+ 85us) client_negotiation.cc:769] Sending connection context
0812 02:30:18.731524 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 02:30:18.731775 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":445,"spinlock_wait_cycles":4992,"thread_start_us":606,"threads_started":1}
I20260812 02:30:18.732419 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.721670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:46785 (local address 127.12.8.194:40267)
0812 02:30:18.722890 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:18.722919 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:18.723183 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:18.723612 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:18.723617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:18.723656 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:18.724389 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.724401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.725973 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.725978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.729072 (+ 3094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:18.729101 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.730965 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.730975 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:18.731306 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.731394 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 02:30:18.731528 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 02:30:18.731775 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":603,"spinlock_wait_cycles":52352,"thread_start_us":719,"threads_started":1}
I20260812 02:30:18.733688 21747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453267088384e0db23c85a09935762a" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a46d39421647f991617549a491783c" is_pre_election: true
I20260812 02:30:18.733728 22090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453267088384e0db23c85a09935762a" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422605f8283c4081859176fc8065e3b1" is_pre_election: true
I20260812 02:30:18.734237 21747 raft_consensus.cc:2454] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 0.
I20260812 02:30:18.734323 22090 raft_consensus.cc:2454] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 0.
I20260812 02:30:18.735069 21829 leader_election.cc:302] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a46d39421647f991617549a491783c, c2310e5e04cc47ccbeb16ed9efd51421; no voters:
I20260812 02:30:18.735669 22190 raft_consensus.cc:2781] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:30:18.735836 22190 raft_consensus.cc:482] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:18.735922 22190 raft_consensus.cc:3037] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:18.738552 22190 raft_consensus.cc:504] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.739279 22190 leader_election.cc:288] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 election: Requested vote from peers 22a46d39421647f991617549a491783c (127.12.8.193:43315), 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:18.739986 21747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453267088384e0db23c85a09935762a" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a46d39421647f991617549a491783c"
I20260812 02:30:18.739985 22090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3453267088384e0db23c85a09935762a" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422605f8283c4081859176fc8065e3b1"
I20260812 02:30:18.740262 22090 raft_consensus.cc:3037] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:18.740262 21747 raft_consensus.cc:3037] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:18.743901 22090 raft_consensus.cc:2454] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 1.
I20260812 02:30:18.743901 21747 raft_consensus.cc:2454] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 1.
I20260812 02:30:18.744553 21829 leader_election.cc:302] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22a46d39421647f991617549a491783c, 422605f8283c4081859176fc8065e3b1, c2310e5e04cc47ccbeb16ed9efd51421; no voters:
I20260812 02:30:18.744974 22190 raft_consensus.cc:2781] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:18.745644 22190 raft_consensus.cc:686] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 LEADER]: Becoming Leader. State: Replica: c2310e5e04cc47ccbeb16ed9efd51421, State: Running, Role: LEADER
I20260812 02:30:18.746260 22190 consensus_queue.cc:237] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.751498 21557 catalog_manager.cc:5495] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20260812 02:30:18.781785 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:18.775950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.12.8.194:34963)
0812 02:30:18.776268 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:18.776276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:18.776332 (+ 56us) server_negotiation.cc:407] Connection header received
0812 02:30:18.776651 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:18.776656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:18.776824 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:18.776977 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:18.777433 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.778200 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.779484 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:18.779910 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:18.780023 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:18.780519 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:18.781115 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:18.781305 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:18.781387 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:30:18.781477 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260812 02:30:18.791927 22090 raft_consensus.cc:1261] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Refusing update from remote peer c2310e5e04cc47ccbeb16ed9efd51421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:18.791927 21747 raft_consensus.cc:1261] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Refusing update from remote peer c2310e5e04cc47ccbeb16ed9efd51421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:18.793196 22195 consensus_queue.cc:1035] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:18.793656 22190 consensus_queue.cc:1035] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:30:18.803128 21974 tablet.cc:2369] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:30:18.851406 21557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:18.851853 21557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:18.857360 21557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: Sending DeleteTablet for 3 replicas of tablet 3453267088384e0db23c85a09935762a
I20260812 02:30:18.859154 22064 tablet_service.cc:1514] Processing DeleteTablet for tablet 3453267088384e0db23c85a09935762a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:18 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:60590
I20260812 02:30:18.859426 21722 tablet_service.cc:1514] Processing DeleteTablet for tablet 3453267088384e0db23c85a09935762a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:18 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:47736
I20260812 02:30:18.860134 21889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3453267088384e0db23c85a09935762a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:18 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51604
I20260812 02:30:18.860963 22213 tablet_replica.cc:330] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: stopping tablet replica
I20260812 02:30:18.861565 22213 raft_consensus.cc:2229] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:18.862161 22213 raft_consensus.cc:2258] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:18.863006 22215 tablet_replica.cc:330] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: stopping tablet replica
I20260812 02:30:18.863018 21557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:30:18.863476 22216 tablet_replica.cc:330] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: stopping tablet replica
I20260812 02:30:18.863410 22215 raft_consensus.cc:2229] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:18.864004 22215 raft_consensus.cc:2258] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 02:30:18.864166 21557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:30:18.864185 22216 raft_consensus.cc:2229] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:30:18.865043 22216 raft_consensus.cc:2258] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:18.866532 22215 ts_tablet_manager.cc:1905] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:18.867548 22213 ts_tablet_manager.cc:1905] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:18.874226 22215 ts_tablet_manager.cc:1918] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:18.874403 22215 log.cc:1196] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/3453267088384e0db23c85a09935762a
I20260812 02:30:18.874879 22213 ts_tablet_manager.cc:1918] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:18.875133 22213 log.cc:1196] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/3453267088384e0db23c85a09935762a
I20260812 02:30:18.875765 22213 ts_tablet_manager.cc:1939] T 3453267088384e0db23c85a09935762a P 422605f8283c4081859176fc8065e3b1: Deleting consensus metadata
I20260812 02:30:18.875666 22064 tablet_service.cc:1467] Processing CreateTablet for tablet b2acdda0074f4c8dbafcf568373c39a5 (DEFAULT_TABLE table=test-table [id=e8fa135559114016adc16fcfcf0aebbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:18.876209 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2acdda0074f4c8dbafcf568373c39a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:18.876096 21722 tablet_service.cc:1467] Processing CreateTablet for tablet b2acdda0074f4c8dbafcf568373c39a5 (DEFAULT_TABLE table=test-table [id=e8fa135559114016adc16fcfcf0aebbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:18.876724 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2acdda0074f4c8dbafcf568373c39a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:18.878115 21537 catalog_manager.cc:4841] TS 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785): tablet 3453267088384e0db23c85a09935762a (table test-table [id=672d5056339e4a64bca8deebe1520ecf]) successfully deleted
I20260812 02:30:18.878953 22215 ts_tablet_manager.cc:1939] T 3453267088384e0db23c85a09935762a P 22a46d39421647f991617549a491783c: Deleting consensus metadata
I20260812 02:30:18.880983 21539 catalog_manager.cc:4841] TS 22a46d39421647f991617549a491783c (127.12.8.193:43315): tablet 3453267088384e0db23c85a09935762a (table test-table [id=672d5056339e4a64bca8deebe1520ecf]) successfully deleted
I20260812 02:30:18.881415 21889 tablet_service.cc:1467] Processing CreateTablet for tablet b2acdda0074f4c8dbafcf568373c39a5 (DEFAULT_TABLE table=test-table [id=e8fa135559114016adc16fcfcf0aebbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:18.881960 21889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2acdda0074f4c8dbafcf568373c39a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:18.882346 22183 tablet_bootstrap.cc:492] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: Bootstrap starting.
I20260812 02:30:18.883985 22183 tablet_bootstrap.cc:654] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:18.884791 22179 tablet_bootstrap.cc:492] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: Bootstrap starting.
I20260812 02:30:18.886772 22179 tablet_bootstrap.cc:654] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:18.891376 22182 tablet_bootstrap.cc:492] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: Bootstrap starting.
I20260812 02:30:18.892141 22216 ts_tablet_manager.cc:1905] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:18.893893 22182 tablet_bootstrap.cc:654] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:18.900549 22179 tablet_bootstrap.cc:492] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: No bootstrap required, opened a new log
I20260812 02:30:18.900743 22179 ts_tablet_manager.cc:1397] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.011s
I20260812 02:30:18.900866 22216 ts_tablet_manager.cc:1918] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:18.901202 22216 log.cc:1196] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/3453267088384e0db23c85a09935762a
I20260812 02:30:18.901266 22183 tablet_bootstrap.cc:492] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: No bootstrap required, opened a new log
I20260812 02:30:18.901564 22183 ts_tablet_manager.cc:1397] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.000s
I20260812 02:30:18.902081 22216 ts_tablet_manager.cc:1939] T 3453267088384e0db23c85a09935762a P c2310e5e04cc47ccbeb16ed9efd51421: Deleting consensus metadata
I20260812 02:30:18.902770 22183 raft_consensus.cc:348] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.903059 22183 raft_consensus.cc:374] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:18.903182 22183 raft_consensus.cc:729] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422605f8283c4081859176fc8065e3b1, State: Initialized, Role: FOLLOWER
I20260812 02:30:18.903097 22179 raft_consensus.cc:348] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.903363 22179 raft_consensus.cc:374] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:18.903470 22179 raft_consensus.cc:729] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a46d39421647f991617549a491783c, State: Initialized, Role: FOLLOWER
I20260812 02:30:18.903544 22183 consensus_queue.cc:260] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.903831 22179 consensus_queue.cc:260] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.905144 22179 ts_tablet_manager.cc:1428] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260812 02:30:18.905579 21539 catalog_manager.cc:4841] TS c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963): tablet 3453267088384e0db23c85a09935762a (table test-table [id=672d5056339e4a64bca8deebe1520ecf]) successfully deleted
I20260812 02:30:18.905635 22183 ts_tablet_manager.cc:1428] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:30:18.906896 22182 tablet_bootstrap.cc:492] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: No bootstrap required, opened a new log
I20260812 02:30:18.907095 22182 ts_tablet_manager.cc:1397] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20260812 02:30:18.908842 22182 raft_consensus.cc:348] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.909127 22182 raft_consensus.cc:374] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:18.909297 22182 raft_consensus.cc:729] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2310e5e04cc47ccbeb16ed9efd51421, State: Initialized, Role: FOLLOWER
I20260812 02:30:18.909601 22182 consensus_queue.cc:260] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.910339 22182 ts_tablet_manager.cc:1428] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:30:18.970113 22210 raft_consensus.cc:482] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:18.970399 22210 raft_consensus.cc:504] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.971530 22210 leader_election.cc:288] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a46d39421647f991617549a491783c (127.12.8.193:43315), 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:18.972327 21747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2acdda0074f4c8dbafcf568373c39a5" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a46d39421647f991617549a491783c" is_pre_election: true
I20260812 02:30:18.972424 22090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2acdda0074f4c8dbafcf568373c39a5" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422605f8283c4081859176fc8065e3b1" is_pre_election: true
I20260812 02:30:18.972599 21747 raft_consensus.cc:2454] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 0.
I20260812 02:30:18.972803 22090 raft_consensus.cc:2454] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 0.
I20260812 02:30:18.973109 21829 leader_election.cc:302] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a46d39421647f991617549a491783c, c2310e5e04cc47ccbeb16ed9efd51421; no voters:
I20260812 02:30:18.973553 22210 raft_consensus.cc:2781] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:30:18.973719 22210 raft_consensus.cc:482] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:18.973824 22210 raft_consensus.cc:3037] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:18.977763 22210 raft_consensus.cc:504] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.978636 22210 leader_election.cc:288] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 election: Requested vote from peers 22a46d39421647f991617549a491783c (127.12.8.193:43315), 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:18.979277 21747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2acdda0074f4c8dbafcf568373c39a5" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a46d39421647f991617549a491783c"
I20260812 02:30:18.979564 21747 raft_consensus.cc:3037] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:18.981456 22090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2acdda0074f4c8dbafcf568373c39a5" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422605f8283c4081859176fc8065e3b1"
I20260812 02:30:18.981715 22090 raft_consensus.cc:3037] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:18.983762 21747 raft_consensus.cc:2454] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 1.
I20260812 02:30:18.984257 22090 raft_consensus.cc:2454] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 1.
I20260812 02:30:18.984549 21829 leader_election.cc:302] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a46d39421647f991617549a491783c, c2310e5e04cc47ccbeb16ed9efd51421; no voters:
I20260812 02:30:18.985029 22210 raft_consensus.cc:2781] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:18.985250 22210 raft_consensus.cc:686] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 LEADER]: Becoming Leader. State: Replica: c2310e5e04cc47ccbeb16ed9efd51421, State: Running, Role: LEADER
I20260812 02:30:18.985587 22210 consensus_queue.cc:237] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:18.988883 21555 catalog_manager.cc:5495] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:30:19 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501816, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.254@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:19.042210 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.024299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.12.8.254:45337)
0812 02:30:19.025214 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.025221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.025520 (+ 299us) server_negotiation.cc:407] Connection header received
0812 02:30:19.025710 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.025714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.025833 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.025931 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:19.027251 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.028103 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.029371 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.029800 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.029911 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.034742 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:19.034748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:19.034753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:19.034826 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:19.038094 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:19.039508 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:19.039517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:19.039520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:19.039711 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:19.040240 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:19.040243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:19.040245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:19.040598 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:19.040726 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.041445 (+ 719us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.041866 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":717,"thread_start_us":451,"threads_started":1}
I20260812 02:30:19.042292 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.024351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45337 (local address 127.0.0.1:56194)
0812 02:30:19.025381 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:19.025403 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:19.025587 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:19.025988 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:19.025993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:19.026808 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:19.027089 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.027100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.028238 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.028242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.029233 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.029247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.030001 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.030007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.030178 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.031085 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:19.031104 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:19.034405 (+ 3301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:19.038369 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:19.038396 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:19.038408 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:19.039259 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:19.039850 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:19.039855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:19.039860 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:19.040093 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:19.040747 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:19.040758 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:19.041033 (+ 275us) client_negotiation.cc:769] Sending connection context
0812 02:30:19.041296 (+ 263us) client_negotiation.cc:240] Negotiation successful
0812 02:30:19.041705 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":688,"threads_started":1}
I20260812 02:30:19.049432 21555 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:56194:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:30:19.051043 21555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:30:19.060072 22064 tablet_service.cc:1467] Processing CreateTablet for tablet f399094b4df14fb3920bd7d8bc7d30d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79cb6ca5018c4d3ca261d06ff382bff7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:30:19.060055 21889 tablet_service.cc:1467] Processing CreateTablet for tablet f399094b4df14fb3920bd7d8bc7d30d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79cb6ca5018c4d3ca261d06ff382bff7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:30:19.060057 21722 tablet_service.cc:1467] Processing CreateTablet for tablet f399094b4df14fb3920bd7d8bc7d30d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79cb6ca5018c4d3ca261d06ff382bff7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:30:19.060735 21889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f399094b4df14fb3920bd7d8bc7d30d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.060721 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f399094b4df14fb3920bd7d8bc7d30d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.060799 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f399094b4df14fb3920bd7d8bc7d30d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.069351 22183 tablet_bootstrap.cc:492] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: Bootstrap starting.
I20260812 02:30:19.069639 22182 tablet_bootstrap.cc:492] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: Bootstrap starting.
I20260812 02:30:19.071146 22183 tablet_bootstrap.cc:654] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.071720 22182 tablet_bootstrap.cc:654] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.073797 22179 tablet_bootstrap.cc:492] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: Bootstrap starting.
I20260812 02:30:19.075877 22179 tablet_bootstrap.cc:654] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.077872 22182 tablet_bootstrap.cc:492] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: No bootstrap required, opened a new log
I20260812 02:30:19.078099 22182 ts_tablet_manager.cc:1397] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260812 02:30:19.079082 22182 raft_consensus.cc:348] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.079393 22182 raft_consensus.cc:374] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.079487 22182 raft_consensus.cc:729] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2310e5e04cc47ccbeb16ed9efd51421, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.079811 22182 consensus_queue.cc:260] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.080549 22182 ts_tablet_manager.cc:1428] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 02:30:19.081523 22210 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.081988 22210 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:30:19.091218 22183 tablet_bootstrap.cc:492] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: No bootstrap required, opened a new log
I20260812 02:30:19.091475 22183 ts_tablet_manager.cc:1397] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.001s
I20260812 02:30:19.092407 22183 raft_consensus.cc:348] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.092690 22183 raft_consensus.cc:374] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.092785 22183 raft_consensus.cc:729] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422605f8283c4081859176fc8065e3b1, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.092789 22179 tablet_bootstrap.cc:492] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: No bootstrap required, opened a new log
I20260812 02:30:19.093012 22179 ts_tablet_manager.cc:1397] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.009s
I20260812 02:30:19.093115 22183 consensus_queue.cc:260] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.093747 22183 ts_tablet_manager.cc:1428] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 02:30:19.093806 22189 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.094267 22189 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:30:19.095062 22179 raft_consensus.cc:348] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.095348 22179 raft_consensus.cc:374] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.095420 22179 raft_consensus.cc:729] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a46d39421647f991617549a491783c, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.095709 22179 consensus_queue.cc:260] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.096422 22179 ts_tablet_manager.cc:1428] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 02:30:19.096347 22192 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.096865 22192 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:30:19.137357 22192 raft_consensus.cc:482] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:19.137609 22192 raft_consensus.cc:504] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.139405 22192 leader_election.cc:288] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963), 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:19.148216 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.139988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:46785 (local address 127.12.8.193:59901)
0812 02:30:19.141105 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:19.141120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:19.141323 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:19.142307 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:19.142312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:19.142341 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:19.142708 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.142720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.144375 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.144381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.146121 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.146133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.147383 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.147389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.147579 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.147619 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 02:30:19.147722 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 02:30:19.147858 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":543,"threads_started":1}
I20260812 02:30:19.148375 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.140152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:59901 (local address 127.12.8.195:46785)
0812 02:30:19.140367 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.140375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.141222 (+ 847us) server_negotiation.cc:407] Connection header received
0812 02:30:19.141470 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.141475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.141559 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.141615 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:19.142949 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.144170 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.146305 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.147149 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.147299 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.147380 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.147805 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.148023 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:30:19.148450 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.140472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:57851 (local address 127.12.8.194:34963)
0812 02:30:19.140708 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.140715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.141034 (+ 319us) server_negotiation.cc:407] Connection header received
0812 02:30:19.141405 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.141412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.141532 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.141612 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:19.142498 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.143427 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.145743 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.147101 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.147336 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.147425 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.147902 (+ 477us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.148097 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 02:30:19.149122 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.139871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:34963 (local address 127.12.8.193:57851)
0812 02:30:19.140881 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:19.140904 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:19.141200 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:19.141854 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:19.141859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:19.141887 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:19.142302 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.142314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.143576 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.143580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.145516 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.145532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.147543 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.147548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.147706 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.147734 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:30:19.148657 (+ 923us) client_negotiation.cc:240] Negotiation successful
0812 02:30:19.148817 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":1872,"threads_started":1}
I20260812 02:30:19.149091 22090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f399094b4df14fb3920bd7d8bc7d30d2" candidate_uuid: "22a46d39421647f991617549a491783c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422605f8283c4081859176fc8065e3b1" is_pre_election: true
I20260812 02:30:19.149482 22090 raft_consensus.cc:2454] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22a46d39421647f991617549a491783c in term 0.
I20260812 02:30:19.150056 21664 leader_election.cc:302] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a46d39421647f991617549a491783c, 422605f8283c4081859176fc8065e3b1; no voters:
I20260812 02:30:19.150462 22192 raft_consensus.cc:2781] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:30:19.150225 21917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f399094b4df14fb3920bd7d8bc7d30d2" candidate_uuid: "22a46d39421647f991617549a491783c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" is_pre_election: true
I20260812 02:30:19.150578 22192 raft_consensus.cc:482] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:19.150665 22192 raft_consensus.cc:3037] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:19.150785 21917 raft_consensus.cc:2454] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22a46d39421647f991617549a491783c in term 0.
I20260812 02:30:19.153281 22192 raft_consensus.cc:504] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.153905 22192 leader_election.cc:288] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [CANDIDATE]: Term 1 election: Requested vote from peers c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963), 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:19.154567 21917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f399094b4df14fb3920bd7d8bc7d30d2" candidate_uuid: "22a46d39421647f991617549a491783c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2310e5e04cc47ccbeb16ed9efd51421"
I20260812 02:30:19.154542 22090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f399094b4df14fb3920bd7d8bc7d30d2" candidate_uuid: "22a46d39421647f991617549a491783c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422605f8283c4081859176fc8065e3b1"
I20260812 02:30:19.154812 21917 raft_consensus.cc:3037] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:19.154842 22090 raft_consensus.cc:3037] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:19.157784 21917 raft_consensus.cc:2454] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22a46d39421647f991617549a491783c in term 1.
I20260812 02:30:19.158253 21666 leader_election.cc:302] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a46d39421647f991617549a491783c, c2310e5e04cc47ccbeb16ed9efd51421; no voters:
I20260812 02:30:19.158593 22192 raft_consensus.cc:2781] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:19.158571 22090 raft_consensus.cc:2454] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22a46d39421647f991617549a491783c in term 1.
I20260812 02:30:19.159915 22192 raft_consensus.cc:686] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [term 1 LEADER]: Becoming Leader. State: Replica: 22a46d39421647f991617549a491783c, State: Running, Role: LEADER
I20260812 02:30:19.160427 22192 consensus_queue.cc:237] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.163570 22236 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: TxnStatusTablet state changed. Reason: New leader 22a46d39421647f991617549a491783c. Latest consensus state: current_term: 1 leader_uuid: "22a46d39421647f991617549a491783c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.164408 22236 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:30:19.165361 22238 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:30:19.165032 21557 catalog_manager.cc:5495] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c reported cstate change: term changed from 0 to 1, leader changed from <none> to 22a46d39421647f991617549a491783c (127.12.8.193). New cstate: current_term: 1 leader_uuid: "22a46d39421647f991617549a491783c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20260812 02:30:19.165616 22238 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:30:19.198952 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.189478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:43315 (local address 127.0.0.1:47742)
0812 02:30:19.189826 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:19.189855 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:19.190055 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:19.191184 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:19.191190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:19.191205 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:30:19.191470 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.191484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.193311 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.193318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.195018 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.195039 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.196321 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.196329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.196551 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.196656 (+ 105us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:30:19.198225 (+ 1569us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:30:19.198267 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 02:30:19.198388 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 02:30:19.198601 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 02:30:19.199069 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.190205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.12.8.193:43315)
0812 02:30:19.190452 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.190459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.190490 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:30:19.190642 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.190649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.190831 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.191109 (+ 278us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:19.191684 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.192990 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.195258 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.196060 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.196249 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.196956 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:19.198048 (+ 1092us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:19.198249 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.198454 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.198585 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 02:30:19.205792 22090 raft_consensus.cc:1261] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Refusing update from remote peer 22a46d39421647f991617549a491783c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:19.207103 21917 raft_consensus.cc:1261] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Refusing update from remote peer 22a46d39421647f991617549a491783c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:19.208010 22236 consensus_queue.cc:1035] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [LEADER]: Connected to new peer: Peer: permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:19.208882 22192 consensus_queue.cc:1035] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:19.224576 22189 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: TxnStatusTablet state changed. Reason: New leader 22a46d39421647f991617549a491783c. Latest consensus state: current_term: 1 leader_uuid: "22a46d39421647f991617549a491783c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.224960 22189 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:30:19.230684 22192 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22a46d39421647f991617549a491783c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.231123 22192 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:30:19.231369 22203 mvcc.cc:205] Tried to move back new op lower bound from 7317511451455987712 to 7317511451286302720. Current Snapshot: MvccSnapshot[applied={T|T < 7317511451455987712}]
I20260812 02:30:19.231454 22210 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: TxnStatusTablet state changed. Reason: New leader 22a46d39421647f991617549a491783c. Latest consensus state: current_term: 1 leader_uuid: "22a46d39421647f991617549a491783c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.231875 22210 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:30:19.236929 22202 mvcc.cc:205] Tried to move back new op lower bound from 7317511451455987712 to 7317511451286302720. Current Snapshot: MvccSnapshot[applied={T|T < 7317511451455987712}]
I20260812 02:30:19.237380 22189 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22a46d39421647f991617549a491783c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.237668 22189 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P 422605f8283c4081859176fc8065e3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:30:19.251523 22236 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22a46d39421647f991617549a491783c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.251963 22236 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P 22a46d39421647f991617549a491783c: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:30:19.255044 22199 mvcc.cc:205] Tried to move back new op lower bound from 7317511451455987712 to 7317511451286302720. Current Snapshot: MvccSnapshot[applied={T|T < 7317511451455987712}]
I20260812 02:30:19.257591 22210 tablet_replica.cc:439] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22a46d39421647f991617549a491783c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } } }
I20260812 02:30:19.258000 22210 tablet_replica.cc:442] T f399094b4df14fb3920bd7d8bc7d30d2 P c2310e5e04cc47ccbeb16ed9efd51421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:30:19.282116 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.272471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:52055 (local address 127.12.8.193:43315)
0812 02:30:19.272710 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.272717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.273516 (+ 799us) server_negotiation.cc:407] Connection header received
0812 02:30:19.273963 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.273980 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.274114 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.274256 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:19.275315 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.276635 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.279405 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.280121 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.280305 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.281140 (+ 835us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:19.281353 (+ 213us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:19.281460 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.281681 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.281779 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 02:30:19.282701 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.272440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:43315 (local address 127.12.8.194:52055)
0812 02:30:19.273382 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:19.273407 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:19.273674 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:19.274594 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:19.274601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:19.274618 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:30:19.275073 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.275091 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.276855 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.276863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.279161 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.279188 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.280427 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.280437 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.280699 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.280827 (+ 128us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:30:19.281487 (+ 660us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:30:19.281504 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 02:30:19.281637 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 02:30:19.281839 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":471,"threads_started":1}
I20260812 02:30:19.300009 22089 raft_consensus.cc:1261] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Refusing update from remote peer c2310e5e04cc47ccbeb16ed9efd51421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:19.300196 21747 raft_consensus.cc:1261] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Refusing update from remote peer c2310e5e04cc47ccbeb16ed9efd51421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:19.300715 22210 consensus_queue.cc:1035] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:19.301378 22210 consensus_queue.cc:1035] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:19.400655 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.379498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:42779 (local address 127.12.8.194:34963)
0812 02:30:19.379711 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.379718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.391922 (+ 12204us) server_negotiation.cc:407] Connection header received
0812 02:30:19.392761 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.392770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.392899 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.392981 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:19.394069 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.395459 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.397829 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.398614 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.398728 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.399644 (+ 916us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:19.399824 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:19.399901 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.400140 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.400256 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:30:19.401823 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.379508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:34963 (local address 127.12.8.193:42779)
0812 02:30:19.391715 (+ 12207us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:19.391748 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:19.392469 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:19.393316 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:19.393322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:19.393339 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:30:19.393853 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.393875 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.395679 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.395685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.397616 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.397638 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.398992 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.398999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.399220 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.399333 (+ 113us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:30:19.399976 (+ 643us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:30:19.399989 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 02:30:19.401042 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0812 02:30:19.401280 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11834,"thread_start_us":12440,"threads_started":1}
I20260812 02:30:19.467670 21557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:19.468057 21557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:19.472003 21557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: Sending DeleteTablet for 3 replicas of tablet b2acdda0074f4c8dbafcf568373c39a5
I20260812 02:30:19.473385 21722 tablet_service.cc:1514] Processing DeleteTablet for tablet b2acdda0074f4c8dbafcf568373c39a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:19 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:47736
I20260812 02:30:19.473436 22064 tablet_service.cc:1514] Processing DeleteTablet for tablet b2acdda0074f4c8dbafcf568373c39a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:19 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:60590
I20260812 02:30:19.474184 21889 tablet_service.cc:1514] Processing DeleteTablet for tablet b2acdda0074f4c8dbafcf568373c39a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:19 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51604
I20260812 02:30:19.478293 22261 tablet_replica.cc:330] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: stopping tablet replica
I20260812 02:30:19.478616 22261 raft_consensus.cc:2229] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:30:19.479156 22261 raft_consensus.cc:2258] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:19.479882 22259 tablet_replica.cc:330] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: stopping tablet replica
I20260812 02:30:19.480099 22259 raft_consensus.cc:2229] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:19.480388 22259 raft_consensus.cc:2258] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:19.481719 22261 ts_tablet_manager.cc:1905] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:19.485123 22259 ts_tablet_manager.cc:1905] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:19.486888 22258 tablet_replica.cc:330] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: stopping tablet replica
I20260812 02:30:19.487142 22258 raft_consensus.cc:2229] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:19.487479 22258 raft_consensus.cc:2258] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:19.488809 22258 ts_tablet_manager.cc:1905] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:19.489049 22261 ts_tablet_manager.cc:1918] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:30:19.489272 22261 log.cc:1196] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/b2acdda0074f4c8dbafcf568373c39a5
I20260812 02:30:19.489796 22261 ts_tablet_manager.cc:1939] T b2acdda0074f4c8dbafcf568373c39a5 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting consensus metadata
I20260812 02:30:19.491479 21539 catalog_manager.cc:4841] TS c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963): tablet b2acdda0074f4c8dbafcf568373c39a5 (table test-table [id=e8fa135559114016adc16fcfcf0aebbf]) successfully deleted
I20260812 02:30:19.494697 22259 ts_tablet_manager.cc:1918] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:30:19.495023 22259 log.cc:1196] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/b2acdda0074f4c8dbafcf568373c39a5
I20260812 02:30:19.495682 22259 ts_tablet_manager.cc:1939] T b2acdda0074f4c8dbafcf568373c39a5 P 22a46d39421647f991617549a491783c: Deleting consensus metadata
I20260812 02:30:19.496037 22258 ts_tablet_manager.cc:1918] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:30:19.496201 22258 log.cc:1196] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/b2acdda0074f4c8dbafcf568373c39a5
I20260812 02:30:19.496647 22258 ts_tablet_manager.cc:1939] T b2acdda0074f4c8dbafcf568373c39a5 P 422605f8283c4081859176fc8065e3b1: Deleting consensus metadata
I20260812 02:30:19.497550 21539 catalog_manager.cc:4841] TS 22a46d39421647f991617549a491783c (127.12.8.193:43315): tablet b2acdda0074f4c8dbafcf568373c39a5 (table test-table [id=e8fa135559114016adc16fcfcf0aebbf]) successfully deleted
I20260812 02:30:19.497908 21537 catalog_manager.cc:4841] TS 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785): tablet b2acdda0074f4c8dbafcf568373c39a5 (table test-table [id=e8fa135559114016adc16fcfcf0aebbf]) successfully deleted
Aug 12 02:30:19 dist-test-slave-vq2q krb5kdc[21500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:30:19 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:19 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:30:19.563861 22269 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:19.550141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.12.8.254:45337)
0812 02:30:19.550942 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.550948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.550999 (+ 51us) server_negotiation.cc:407] Connection header received
0812 02:30:19.551060 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.551065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.551138 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.551230 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:19.553269 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.554117 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.555982 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.556522 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.556630 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.563521 (+ 6891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56210: BlockingRecv error: recv got EOF from 127.0.0.1:56210 (error 108)
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":527,"threads_started":1}
Aug 12 02:30:19 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM
I20260812 02:30:19.586652 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.568179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.12.8.254:45337)
0812 02:30:19.568635 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.568642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.569242 (+ 600us) server_negotiation.cc:407] Connection header received
0812 02:30:19.569398 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.569405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.569519 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.569697 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:19.571066 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.572321 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.573870 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.574517 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.574701 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.579352 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:19.579361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:19.579368 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:19.579434 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:19.583356 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:19.584213 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:19.584221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:19.584224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:19.584424 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:19.584869 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:19.584874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:19.584878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:19.585357 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:19.585663 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.585974 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.586252 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260812 02:30:19.591673 21557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:30:19.592391 21557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:30:19.600468 21889 tablet_service.cc:1467] Processing CreateTablet for tablet 794986f288514ccd8dfcc943e617e338 (DEFAULT_TABLE table=test-table [id=62f23875053a44a2be79b53e5300d14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:19.600613 22064 tablet_service.cc:1467] Processing CreateTablet for tablet 794986f288514ccd8dfcc943e617e338 (DEFAULT_TABLE table=test-table [id=62f23875053a44a2be79b53e5300d14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:19.600679 21722 tablet_service.cc:1467] Processing CreateTablet for tablet 794986f288514ccd8dfcc943e617e338 (DEFAULT_TABLE table=test-table [id=62f23875053a44a2be79b53e5300d14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:19.601001 21889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794986f288514ccd8dfcc943e617e338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.601018 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794986f288514ccd8dfcc943e617e338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.601408 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794986f288514ccd8dfcc943e617e338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.614310 22275 tablet_bootstrap.cc:492] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: Bootstrap starting.
I20260812 02:30:19.616137 22277 tablet_bootstrap.cc:492] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: Bootstrap starting.
I20260812 02:30:19.616587 22275 tablet_bootstrap.cc:654] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.616544 22278 tablet_bootstrap.cc:492] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: Bootstrap starting.
I20260812 02:30:19.618079 22277 tablet_bootstrap.cc:654] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.623926 22277 tablet_bootstrap.cc:492] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: No bootstrap required, opened a new log
I20260812 02:30:19.623989 22278 tablet_bootstrap.cc:654] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.624106 22277 ts_tablet_manager.cc:1397] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 02:30:19.625013 22277 raft_consensus.cc:348] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.625310 22277 raft_consensus.cc:374] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.625407 22277 raft_consensus.cc:729] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422605f8283c4081859176fc8065e3b1, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.625854 22277 consensus_queue.cc:260] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.626211 22189 raft_consensus.cc:482] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:19.626371 22189 raft_consensus.cc:504] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.626773 22275 tablet_bootstrap.cc:492] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: No bootstrap required, opened a new log
I20260812 02:30:19.626971 22275 ts_tablet_manager.cc:1397] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260812 02:30:19.627848 22275 raft_consensus.cc:348] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.628181 22275 raft_consensus.cc:374] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.628248 22189 leader_election.cc:288] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a46d39421647f991617549a491783c (127.12.8.193:43315), c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963)
I20260812 02:30:19.628327 22275 raft_consensus.cc:729] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2310e5e04cc47ccbeb16ed9efd51421, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.628695 22275 consensus_queue.cc:260] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.630023 22275 ts_tablet_manager.cc:1428] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260812 02:30:19.636480 22278 tablet_bootstrap.cc:492] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: No bootstrap required, opened a new log
I20260812 02:30:19.636698 22278 ts_tablet_manager.cc:1397] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20260812 02:30:19.637884 22278 raft_consensus.cc:348] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.638224 22278 raft_consensus.cc:374] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.638332 22278 raft_consensus.cc:729] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a46d39421647f991617549a491783c, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.638676 22278 consensus_queue.cc:260] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.640225 22278 ts_tablet_manager.cc:1428] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260812 02:30:19.642508 22277 ts_tablet_manager.cc:1428] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20260812 02:30:19.654659 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.629441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:40843 (local address 127.12.8.193:43315)
0812 02:30:19.629648 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.629654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.644772 (+ 15118us) server_negotiation.cc:407] Connection header received
0812 02:30:19.645212 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.645221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.645336 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.645419 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:19.646445 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.647662 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.651796 (+ 4134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.653406 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.653595 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.653678 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.654150 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.654309 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 02:30:19.654739 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.628779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:43315 (local address 127.12.8.195:40843)
0812 02:30:19.644607 (+ 15828us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:19.644637 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:19.644942 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:19.645696 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:19.645702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:19.645754 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:19.646218 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.646235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.647903 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.647910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.651478 (+ 3568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.651502 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.653663 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.653671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.653890 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.653940 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 02:30:19.654099 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 02:30:19.654291 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15657,"thread_start_us":15451,"threads_started":1}
I20260812 02:30:19.655485 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.645600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:45641 (local address 127.12.8.194:34963)
0812 02:30:19.645897 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.645904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.648640 (+ 2736us) server_negotiation.cc:407] Connection header received
0812 02:30:19.648978 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.648983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.649068 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.649136 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:19.650086 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.651082 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.653277 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.654263 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.654392 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.654470 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.655114 (+ 644us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.655255 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 02:30:19.656273 21747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794986f288514ccd8dfcc943e617e338" candidate_uuid: "422605f8283c4081859176fc8065e3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a46d39421647f991617549a491783c" is_pre_election: true
I20260812 02:30:19.656544 21747 raft_consensus.cc:2454] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422605f8283c4081859176fc8065e3b1 in term 0.
I20260812 02:30:19.657338 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.644952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:34963 (local address 127.12.8.195:45641)
0812 02:30:19.648535 (+ 3583us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:19.648553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:19.648766 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:19.649394 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:19.649398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:19.649424 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:19.649927 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.649941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.651223 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.651229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.653041 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:19.653058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.654697 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.654705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:19.654913 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.654949 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 02:30:19.655949 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0812 02:30:19.656950 (+ 1001us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3459,"thread_start_us":3214,"threads_started":1}
I20260812 02:30:19.657245 22003 leader_election.cc:302] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a46d39421647f991617549a491783c, 422605f8283c4081859176fc8065e3b1; no voters:
I20260812 02:30:19.657801 22189 raft_consensus.cc:2781] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:30:19.657959 22189 raft_consensus.cc:482] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:19.658095 22189 raft_consensus.cc:3037] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:19.661909 21917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794986f288514ccd8dfcc943e617e338" candidate_uuid: "422605f8283c4081859176fc8065e3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" is_pre_election: true
I20260812 02:30:19.662168 21917 raft_consensus.cc:2454] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422605f8283c4081859176fc8065e3b1 in term 0.
I20260812 02:30:19.662127 22189 raft_consensus.cc:504] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.663173 22189 leader_election.cc:288] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [CANDIDATE]: Term 1 election: Requested vote from peers 22a46d39421647f991617549a491783c (127.12.8.193:43315), c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963)
I20260812 02:30:19.663715 21917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794986f288514ccd8dfcc943e617e338" candidate_uuid: "422605f8283c4081859176fc8065e3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2310e5e04cc47ccbeb16ed9efd51421"
I20260812 02:30:19.663884 21917 raft_consensus.cc:3037] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:19.663905 21747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794986f288514ccd8dfcc943e617e338" candidate_uuid: "422605f8283c4081859176fc8065e3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a46d39421647f991617549a491783c"
I20260812 02:30:19.664225 21747 raft_consensus.cc:3037] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:19.666414 21917 raft_consensus.cc:2454] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422605f8283c4081859176fc8065e3b1 in term 1.
I20260812 02:30:19.666872 22003 leader_election.cc:302] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422605f8283c4081859176fc8065e3b1, c2310e5e04cc47ccbeb16ed9efd51421; no voters:
I20260812 02:30:19.668097 21747 raft_consensus.cc:2454] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422605f8283c4081859176fc8065e3b1 in term 1.
I20260812 02:30:19.668663 22189 raft_consensus.cc:2781] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:19.669354 22189 raft_consensus.cc:686] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 1 LEADER]: Becoming Leader. State: Replica: 422605f8283c4081859176fc8065e3b1, State: Running, Role: LEADER
I20260812 02:30:19.669945 22189 consensus_queue.cc:237] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.675412 21557 catalog_manager.cc:5495] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 422605f8283c4081859176fc8065e3b1 (127.12.8.195). New cstate: current_term: 1 leader_uuid: "422605f8283c4081859176fc8065e3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } health_report { overall_health: HEALTHY } } }
I20260812 02:30:19.695192 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:19.688611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.12.8.195:46785)
0812 02:30:19.689585 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:19.689590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:19.689631 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:30:19.689738 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:19.689750 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:19.689840 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:19.689999 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:19.690468 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.691270 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.692608 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:19.693039 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:19.693146 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:19.693675 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:19.694362 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:19.694497 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:19.694617 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 02:30:19.694704 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":647,"threads_started":1}
I20260812 02:30:19.700242 21917 raft_consensus.cc:1261] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Refusing update from remote peer 422605f8283c4081859176fc8065e3b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:19.700572 21747 raft_consensus.cc:1261] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Refusing update from remote peer 422605f8283c4081859176fc8065e3b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:19.701171 22286 consensus_queue.cc:1035] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:19.701622 22189 consensus_queue.cc:1035] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:19.732543 21556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:19.732817 21556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:19.735920 21556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: Sending DeleteTablet for 3 replicas of tablet 794986f288514ccd8dfcc943e617e338
I20260812 02:30:19.737143 22064 tablet_service.cc:1514] Processing DeleteTablet for tablet 794986f288514ccd8dfcc943e617e338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:19 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:60590
I20260812 02:30:19.737221 21722 tablet_service.cc:1514] Processing DeleteTablet for tablet 794986f288514ccd8dfcc943e617e338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:19 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:47736
I20260812 02:30:19.737504 22258 tablet_replica.cc:330] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: stopping tablet replica
I20260812 02:30:19.737579 21889 tablet_service.cc:1514] Processing DeleteTablet for tablet 794986f288514ccd8dfcc943e617e338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:19 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51604
I20260812 02:30:19.737670 22259 tablet_replica.cc:330] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: stopping tablet replica
I20260812 02:30:19.737735 22258 raft_consensus.cc:2229] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:30:19.737989 22261 tablet_replica.cc:330] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: stopping tablet replica
I20260812 02:30:19.738014 22259 raft_consensus.cc:2229] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:19.738245 22258 raft_consensus.cc:2258] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:19.738270 22261 raft_consensus.cc:2229] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:19.738335 22259 raft_consensus.cc:2258] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:19.738572 22261 raft_consensus.cc:2258] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:19.739358 22258 ts_tablet_manager.cc:1905] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:19.740116 21556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:30:19.740792 21556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:30:19.748803 22064 tablet_service.cc:1467] Processing CreateTablet for tablet 4972a8182a8643008a3d5d6a71837eb6 (DEFAULT_TABLE table=test-table [id=d0a1a1b761ac400a86388396c806e946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:19.748803 21889 tablet_service.cc:1467] Processing CreateTablet for tablet 4972a8182a8643008a3d5d6a71837eb6 (DEFAULT_TABLE table=test-table [id=d0a1a1b761ac400a86388396c806e946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:19.749333 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4972a8182a8643008a3d5d6a71837eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.749620 21722 tablet_service.cc:1467] Processing CreateTablet for tablet 4972a8182a8643008a3d5d6a71837eb6 (DEFAULT_TABLE table=test-table [id=d0a1a1b761ac400a86388396c806e946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:19.750128 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4972a8182a8643008a3d5d6a71837eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.753352 21889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4972a8182a8643008a3d5d6a71837eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:19.759711 22277 tablet_bootstrap.cc:492] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: Bootstrap starting.
I20260812 02:30:19.761371 22278 tablet_bootstrap.cc:492] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: Bootstrap starting.
I20260812 02:30:19.762152 22277 tablet_bootstrap.cc:654] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.763480 22278 tablet_bootstrap.cc:654] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.764997 22275 tablet_bootstrap.cc:492] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: Bootstrap starting.
I20260812 02:30:19.766914 22275 tablet_bootstrap.cc:654] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:19.774276 22259 ts_tablet_manager.cc:1905] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:19.775414 22278 tablet_bootstrap.cc:492] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: No bootstrap required, opened a new log
I20260812 02:30:19.775640 22278 ts_tablet_manager.cc:1397] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20260812 02:30:19.776571 22278 raft_consensus.cc:348] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.776926 22278 raft_consensus.cc:374] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.777014 22278 raft_consensus.cc:729] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a46d39421647f991617549a491783c, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.777355 22278 consensus_queue.cc:260] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.778148 22278 ts_tablet_manager.cc:1428] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 02:30:19.788422 22259 ts_tablet_manager.cc:1918] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:19.788969 22259 log.cc:1196] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/794986f288514ccd8dfcc943e617e338
I20260812 02:30:19.789453 22259 ts_tablet_manager.cc:1939] T 794986f288514ccd8dfcc943e617e338 P 22a46d39421647f991617549a491783c: Deleting consensus metadata
I20260812 02:30:19.790591 21539 catalog_manager.cc:4841] TS 22a46d39421647f991617549a491783c (127.12.8.193:43315): tablet 794986f288514ccd8dfcc943e617e338 (table test-table [id=62f23875053a44a2be79b53e5300d14b]) successfully deleted
I20260812 02:30:19.791353 22261 ts_tablet_manager.cc:1905] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:19.794204 22258 ts_tablet_manager.cc:1918] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:19.795197 22258 log.cc:1196] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/794986f288514ccd8dfcc943e617e338
I20260812 02:30:19.795913 22258 ts_tablet_manager.cc:1939] T 794986f288514ccd8dfcc943e617e338 P 422605f8283c4081859176fc8065e3b1: Deleting consensus metadata
I20260812 02:30:19.797673 21537 catalog_manager.cc:4841] TS 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785): tablet 794986f288514ccd8dfcc943e617e338 (table test-table [id=62f23875053a44a2be79b53e5300d14b]) successfully deleted
I20260812 02:30:19.798677 22261 ts_tablet_manager.cc:1918] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:19.798847 22261 log.cc:1196] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/794986f288514ccd8dfcc943e617e338
I20260812 02:30:19.799263 22261 ts_tablet_manager.cc:1939] T 794986f288514ccd8dfcc943e617e338 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting consensus metadata
I20260812 02:30:19.800367 21539 catalog_manager.cc:4841] TS c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963): tablet 794986f288514ccd8dfcc943e617e338 (table test-table [id=62f23875053a44a2be79b53e5300d14b]) successfully deleted
I20260812 02:30:19.803680 22275 tablet_bootstrap.cc:492] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: No bootstrap required, opened a new log
I20260812 02:30:19.803972 22275 ts_tablet_manager.cc:1397] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.004s
I20260812 02:30:19.804826 22275 raft_consensus.cc:348] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.805107 22275 raft_consensus.cc:374] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.805226 22275 raft_consensus.cc:729] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2310e5e04cc47ccbeb16ed9efd51421, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.805522 22275 consensus_queue.cc:260] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.806279 22275 ts_tablet_manager.cc:1428] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 02:30:19.813691 22277 tablet_bootstrap.cc:492] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: No bootstrap required, opened a new log
I20260812 02:30:19.813954 22277 ts_tablet_manager.cc:1397] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: Time spent bootstrapping tablet: real 0.054s user 0.018s sys 0.008s
I20260812 02:30:19.815464 22277 raft_consensus.cc:348] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.815800 22277 raft_consensus.cc:374] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:19.815900 22277 raft_consensus.cc:729] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422605f8283c4081859176fc8065e3b1, State: Initialized, Role: FOLLOWER
I20260812 02:30:19.816202 22277 consensus_queue.cc:260] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:19.816886 22277 ts_tablet_manager.cc:1428] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 02:30:20.093495 22210 raft_consensus.cc:482] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:20.093729 22210 raft_consensus.cc:504] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:20.094504 22210 leader_election.cc:288] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a46d39421647f991617549a491783c (127.12.8.193:43315), 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:20.095357 22089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4972a8182a8643008a3d5d6a71837eb6" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422605f8283c4081859176fc8065e3b1" is_pre_election: true
I20260812 02:30:20.095335 21747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4972a8182a8643008a3d5d6a71837eb6" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a46d39421647f991617549a491783c" is_pre_election: true
I20260812 02:30:20.095713 22089 raft_consensus.cc:2454] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 0.
I20260812 02:30:20.095767 21747 raft_consensus.cc:2454] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 0.
I20260812 02:30:20.096324 21826 leader_election.cc:302] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422605f8283c4081859176fc8065e3b1, c2310e5e04cc47ccbeb16ed9efd51421; no voters:
I20260812 02:30:20.096745 22210 raft_consensus.cc:2781] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:30:20.096836 22210 raft_consensus.cc:482] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:20.096920 22210 raft_consensus.cc:3037] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:20.099059 22210 raft_consensus.cc:504] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:20.099689 22210 leader_election.cc:288] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 election: Requested vote from peers 22a46d39421647f991617549a491783c (127.12.8.193:43315), 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785)
I20260812 02:30:20.100399 22089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4972a8182a8643008a3d5d6a71837eb6" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422605f8283c4081859176fc8065e3b1"
I20260812 02:30:20.100653 22089 raft_consensus.cc:3037] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:20.100498 21747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4972a8182a8643008a3d5d6a71837eb6" candidate_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a46d39421647f991617549a491783c"
I20260812 02:30:20.100908 21747 raft_consensus.cc:3037] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:20.103943 22089 raft_consensus.cc:2454] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 1.
I20260812 02:30:20.104480 21826 leader_election.cc:302] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422605f8283c4081859176fc8065e3b1, c2310e5e04cc47ccbeb16ed9efd51421; no voters:
I20260812 02:30:20.104827 22210 raft_consensus.cc:2781] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:20.104868 21747 raft_consensus.cc:2454] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2310e5e04cc47ccbeb16ed9efd51421 in term 1.
I20260812 02:30:20.105623 22210 raft_consensus.cc:686] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 LEADER]: Becoming Leader. State: Replica: c2310e5e04cc47ccbeb16ed9efd51421, State: Running, Role: LEADER
I20260812 02:30:20.105959 22210 consensus_queue.cc:237] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } }
I20260812 02:30:20.108778 21556 catalog_manager.cc:5495] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2310e5e04cc47ccbeb16ed9efd51421" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20260812 02:30:20.155555 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:20.150463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.12.8.194:34963)
0812 02:30:20.150613 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:20.150618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:20.150928 (+ 310us) server_negotiation.cc:407] Connection header received
0812 02:30:20.151107 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:20.151113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:20.151179 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:20.151254 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:20.151677 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:20.152451 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:20.153907 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:20.154315 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:20.154427 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:20.154850 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:20.154961 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:20.155049 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:20.155190 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 02:30:20.155270 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:30:20.167989 22089 raft_consensus.cc:1261] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Refusing update from remote peer c2310e5e04cc47ccbeb16ed9efd51421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:20.167982 21747 raft_consensus.cc:1261] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Refusing update from remote peer c2310e5e04cc47ccbeb16ed9efd51421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:20.168675 22297 consensus_queue.cc:1035] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "422605f8283c4081859176fc8065e3b1" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:20.169071 22210 consensus_queue.cc:1035] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a46d39421647f991617549a491783c" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:20.251207 21556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:20.251462 21556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:20.254581 21556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0a8bdde77b44e368ad78f90272480b2: Sending DeleteTablet for 3 replicas of tablet 4972a8182a8643008a3d5d6a71837eb6
I20260812 02:30:20.255456 21722 tablet_service.cc:1514] Processing DeleteTablet for tablet 4972a8182a8643008a3d5d6a71837eb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:20 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:47736
I20260812 02:30:20.255463 21889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4972a8182a8643008a3d5d6a71837eb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:20 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:51604
I20260812 02:30:20.255796 22261 tablet_replica.cc:330] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: stopping tablet replica
I20260812 02:30:20.255918 22064 tablet_service.cc:1514] Processing DeleteTablet for tablet 4972a8182a8643008a3d5d6a71837eb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:20 UTC) from {username='oryx', principal='oryx/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:60590
I20260812 02:30:20.255992 22261 raft_consensus.cc:2229] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:30:20.256337 22261 raft_consensus.cc:2258] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:20.256524 22258 tablet_replica.cc:330] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: stopping tablet replica
I20260812 02:30:20.256708 22258 raft_consensus.cc:2229] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:20.256907 22258 raft_consensus.cc:2258] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:20.257678 22259 tablet_replica.cc:330] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: stopping tablet replica
I20260812 02:30:20.257874 22259 raft_consensus.cc:2229] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:20.258114 22259 raft_consensus.cc:2258] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:20.258586 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 21648
I20260812 02:30:20.258620 22261 ts_tablet_manager.cc:1905] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:20.259370 22259 ts_tablet_manager.cc:1905] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:20.268110 22259 ts_tablet_manager.cc:1918] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:30:20.268415 22259 log.cc:1196] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/4972a8182a8643008a3d5d6a71837eb6
I20260812 02:30:20.268488 22261 ts_tablet_manager.cc:1918] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:30:20.268630 22261 log.cc:1196] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/4972a8182a8643008a3d5d6a71837eb6
I20260812 02:30:20.269048 22261 ts_tablet_manager.cc:1939] T 4972a8182a8643008a3d5d6a71837eb6 P c2310e5e04cc47ccbeb16ed9efd51421: Deleting consensus metadata
I20260812 02:30:20.269049 22259 ts_tablet_manager.cc:1939] T 4972a8182a8643008a3d5d6a71837eb6 P 22a46d39421647f991617549a491783c: Deleting consensus metadata
I20260812 02:30:20.270159 22258 ts_tablet_manager.cc:1905] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:20.274566 21539 catalog_manager.cc:4841] TS c2310e5e04cc47ccbeb16ed9efd51421 (127.12.8.194:34963): tablet 4972a8182a8643008a3d5d6a71837eb6 (table test-table [id=d0a1a1b761ac400a86388396c806e946]) successfully deleted
I20260812 02:30:20.278537 22258 ts_tablet_manager.cc:1918] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:30:20.278770 22258 log.cc:1196] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/4972a8182a8643008a3d5d6a71837eb6
I20260812 02:30:20.279304 22258 ts_tablet_manager.cc:1939] T 4972a8182a8643008a3d5d6a71837eb6 P 422605f8283c4081859176fc8065e3b1: Deleting consensus metadata
I20260812 02:30:20.281128 21539 catalog_manager.cc:4841] TS 22a46d39421647f991617549a491783c (127.12.8.193:43315): tablet 4972a8182a8643008a3d5d6a71837eb6 (table test-table [id=d0a1a1b761ac400a86388396c806e946]) successfully deleted
I20260812 02:30:20.282220 21537 catalog_manager.cc:4841] TS 422605f8283c4081859176fc8065e3b1 (127.12.8.195:46785): tablet 4972a8182a8643008a3d5d6a71837eb6 (table test-table [id=d0a1a1b761ac400a86388396c806e946]) successfully deleted
Aug 12 02:30:20 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.193@KRBTEST.COM
I20260812 02:30:20.297209 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:20.262069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.12.8.193:43315)
0812 02:30:20.280390 (+ 18321us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:20.280397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:20.282603 (+ 2206us) server_negotiation.cc:407] Connection header received
0812 02:30:20.282761 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:20.282766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:20.282873 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:20.283012 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:20.284216 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:20.285360 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:20.286595 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:20.287172 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:20.287291 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:20.291388 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:20.291423 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:20.291438 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:20.291472 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:20.294379 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:20.295213 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:20.295220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:20.295222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:20.295365 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:20.295745 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:20.295748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:20.295750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:20.296108 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:20.296348 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:20.296638 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 02:30:20.296828 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18182,"thread_start_us":18337,"threads_started":1}
I20260812 02:30:20.297869 21797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:20.434371 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 21648
I20260812 02:30:20.463204 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 21808
Aug 12 02:30:20 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.194@KRBTEST.COM
I20260812 02:30:20.478813 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:20.463955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.12.8.194:34963)
0812 02:30:20.464166 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:20.464171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:20.464189 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:30:20.464324 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:20.464328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:20.464413 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:20.464498 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:20.465844 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:20.466611 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:20.467848 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:20.468260 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:20.468357 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:20.472750 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:20.472771 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:20.472787 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:20.472835 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:20.475954 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:20.476692 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:20.476699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:20.476701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:20.476848 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:20.477287 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:20.477291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:20.477294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:20.477703 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:20.477973 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:20.478267 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 02:30:20.478447 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 02:30:20.479559 21968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:20.612018 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 21808
I20260812 02:30:20.640231 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 21979
Aug 12 02:30:20 dist-test-slave-vq2q krb5kdc[21500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.195@KRBTEST.COM
I20260812 02:30:20.655858 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:20.641088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.12.8.195:46785)
0812 02:30:20.641566 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:20.641570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:20.641587 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:30:20.641642 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:20.641646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:20.641709 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:20.641815 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:20.643143 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:20.643907 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:20.645255 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:20.645685 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:20.645799 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:20.649995 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:20.650008 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:20.650025 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:20.650058 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:20.652972 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:20.653768 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:20.653775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:20.653777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:20.653939 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:20.654323 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:20.654326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:20.654328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:20.654685 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:20.655009 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:20.655278 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 02:30:20.655514 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":266,"threads_started":1}
I20260812 02:30:20.656603 22147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:20.785157 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 21979
I20260812 02:30:20.812733 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 21516
I20260812 02:30:20.814061 21605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:20.937485 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 21516
2026-08-12T02:30:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6660 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:30:21 dist-test-slave-vq2q krb5kdc[22317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:30:21 dist-test-slave-vq2q krb5kdc[22317](info): set up 2 sockets
Aug 12 02:30:21 dist-test-slave-vq2q krb5kdc[22317](info): commencing operation
krb5kdc: starting...
W20260812 02:30:22.093098 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:30:22 dist-test-slave-vq2q krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:30:22.442013 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:36001
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:36001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:22.877117 22331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:22.877584 22331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:22.877712 22331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:22.886188 22331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:22.886391 22331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:22.886508 22331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:22.886620 22331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:22.886694 22331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:22.898736 22331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:36001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:36001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:22.900564 22331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:22.902336 22331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:22.917943 22345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:22.918097 22339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:22.918175 22340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:22.919756 22331 server_base.cc:1034] running on GCE node
I20260812 02:30:22.920941 22331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:22.923400 22331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:22.924736 22331 hybrid_clock.cc:648] HybridClock initialized: now 1786501822924653 us; error 54 us; skew 500 ppm
I20260812 02:30:22.925345 22331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T02:30:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2063 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:30:23 dist-test-slave-vq2q krb5kdc[22355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:30:23 dist-test-slave-vq2q krb5kdc[22355](info): set up 2 sockets
Aug 12 02:30:23 dist-test-slave-vq2q krb5kdc[22355](info): commencing operation
krb5kdc: starting...
W20260812 02:30:24.162915 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.005s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:30:24 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:30:24.518090 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46453
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:46453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:24.918248 22369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:24.918658 22369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:24.918762 22369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:24.927194 22369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:24.927366 22369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:24.927418 22369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:24.927484 22369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:24.927536 22369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:24.939023 22369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:46453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:24.940789 22369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:24.942482 22369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:24.956933 22375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:24.956961 22379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:24.957041 22376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:24.958107 22369 server_base.cc:1034] running on GCE node
I20260812 02:30:24.959100 22369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:24.960966 22369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:24.962291 22369 hybrid_clock.cc:648] HybridClock initialized: now 1786501824962197 us; error 73 us; skew 500 ppm
I20260812 02:30:24.962770 22369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:24 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501824, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:24.972501 22369 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:30:24.974647 22369 webserver.cc:460] Webserver started at http://127.12.8.254:32865/ using document root <none> and password file <none>
I20260812 02:30:24.975718 22369 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:24.975847 22369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:24.976256 22369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:24.980189 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "d294b912188c4e97a2550dfb62afce6e"
format_stamp: "Formatted at 2026-08-12 02:30:24 on dist-test-slave-vq2q"
server_key: "ed329ab789dff3f6fa91d22cd4cbfc2d"
server_key_iv: "f29cf8725bb7dfbe8317783840838482"
server_key_version: "encryptionkey@0"
I20260812 02:30:24.981271 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "d294b912188c4e97a2550dfb62afce6e"
format_stamp: "Formatted at 2026-08-12 02:30:24 on dist-test-slave-vq2q"
server_key: "ed329ab789dff3f6fa91d22cd4cbfc2d"
server_key_iv: "f29cf8725bb7dfbe8317783840838482"
server_key_version: "encryptionkey@0"
I20260812 02:30:24.988308 22369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 02:30:24.992894 22386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:24.994748 22369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:30:24.994994 22369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "d294b912188c4e97a2550dfb62afce6e"
format_stamp: "Formatted at 2026-08-12 02:30:24 on dist-test-slave-vq2q"
server_key: "ed329ab789dff3f6fa91d22cd4cbfc2d"
server_key_iv: "f29cf8725bb7dfbe8317783840838482"
server_key_version: "encryptionkey@0"
I20260812 02:30:24.995275 22369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:25.024233 22369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:25.029947 22369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:25.030309 22369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:25.055190 22369 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:46453
I20260812 02:30:25.055258 22449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:46453 every 8 connection(s)
I20260812 02:30:25.057216 22369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:30:25.061672 22451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:25.067734 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 22369
I20260812 02:30:25.068001 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:30:25.068785 12323 external_mini_cluster.cc:1428] Setting key c718b09da3f5d9dcd0bbf806fee1d607
I20260812 02:30:25.073395 22451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Bootstrap starting.
I20260812 02:30:25.077118 22451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:25.078473 22451 log.cc:824] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:25.082387 22451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: No bootstrap required, opened a new log
Aug 12 02:30:25 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:25.092499 22451 raft_consensus.cc:348] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } }
I20260812 02:30:25.092960 22451 raft_consensus.cc:374] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:25.093080 22451 raft_consensus.cc:729] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d294b912188c4e97a2550dfb62afce6e, State: Initialized, Role: FOLLOWER
I20260812 02:30:25.093752 22451 consensus_queue.cc:260] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } }
I20260812 02:30:25.094096 22451 raft_consensus.cc:388] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:30:25.094266 22451 raft_consensus.cc:482] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:30:25.094455 22451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:25.097826 22451 raft_consensus.cc:504] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } }
I20260812 02:30:25.098352 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:25.071487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.12.8.254:46453)
0812 02:30:25.072516 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:25.072531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:25.072594 (+ 63us) server_negotiation.cc:407] Connection header received
0812 02:30:25.074873 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:25.074905 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:25.075461 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:25.075896 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:25.078044 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:25.080012 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:25.081902 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:25.082785 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:25.082957 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:25.088276 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:25.088293 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:25.088315 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:25.088369 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:25.092571 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:25.093859 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:25.093869 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:25.093878 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:25.094128 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:25.094667 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:25.094672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:25.094675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:25.095458 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:25.095840 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:25.096353 (+ 513us) server_negotiation.cc:299] Negotiation successful
0812 02:30:25.096865 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":474,"threads_started":1}
I20260812 02:30:25.098471 22451 leader_election.cc:302] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d294b912188c4e97a2550dfb62afce6e; no voters:
I20260812 02:30:25.099238 22451 leader_election.cc:288] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:30:25.099536 22457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:25.100838 22457 raft_consensus.cc:686] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 1 LEADER]: Becoming Leader. State: Replica: d294b912188c4e97a2550dfb62afce6e, State: Running, Role: LEADER
I20260812 02:30:25.101838 22451 sys_catalog.cc:564] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:25.101715 22457 consensus_queue.cc:237] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } }
I20260812 02:30:25.105962 22459 sys_catalog.cc:455] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d294b912188c4e97a2550dfb62afce6e. Latest consensus state: current_term: 1 leader_uuid: "d294b912188c4e97a2550dfb62afce6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } } }
I20260812 02:30:25.106578 22459 sys_catalog.cc:458] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:25.105935 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d294b912188c4e97a2550dfb62afce6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } } }
I20260812 02:30:25.107169 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:25.108275 22469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:30:25.114861 22469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:30:25.126945 22469 catalog_manager.cc:1347] Generated new cluster ID: 05bad871e85b46798ab8bfce84990528
I20260812 02:30:25.127080 22469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:30:25.148128 22469 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:30:25.150842 22469 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:30:25.164639 22469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Generated new TSK 0
I20260812 02:30:25.165668 22469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:30:25.249974 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46453
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:25.685019 22482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:25.685514 22482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:25.685699 22482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:25.694324 22482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:25.694520 22482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:25.694732 22482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:30:25.707062 22482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:25.708936 22482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:25.710747 22482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:25.727890 22491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:25.727993 22488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:25.727780 22489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:25.729226 22482 server_base.cc:1034] running on GCE node
I20260812 02:30:25.730376 22482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:25.731805 22482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:25.733098 22482 hybrid_clock.cc:648] HybridClock initialized: now 1786501825733028 us; error 46 us; skew 500 ppm
I20260812 02:30:25.733652 22482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:25 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501825, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:25.737776 22482 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:30:25.739948 22482 webserver.cc:460] Webserver started at http://127.12.8.193:39371/ using document root <none> and password file <none>
I20260812 02:30:25.740877 22482 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:25.741025 22482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:25.741453 22482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:25.745129 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "fb2f419cf6fd4433b31cd75992e25b01"
format_stamp: "Formatted at 2026-08-12 02:30:25 on dist-test-slave-vq2q"
server_key: "fa9567b2d9688b17fa824f206daf907a"
server_key_iv: "07b87df5cac1d922829f0b843bace8ed"
server_key_version: "encryptionkey@0"
I20260812 02:30:25.746136 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "fb2f419cf6fd4433b31cd75992e25b01"
format_stamp: "Formatted at 2026-08-12 02:30:25 on dist-test-slave-vq2q"
server_key: "fa9567b2d9688b17fa824f206daf907a"
server_key_iv: "07b87df5cac1d922829f0b843bace8ed"
server_key_version: "encryptionkey@0"
I20260812 02:30:25.754994 22482 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260812 02:30:25.761154 22498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:25.763447 22482 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 02:30:25.763753 22482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "fb2f419cf6fd4433b31cd75992e25b01"
format_stamp: "Formatted at 2026-08-12 02:30:25 on dist-test-slave-vq2q"
server_key: "fa9567b2d9688b17fa824f206daf907a"
server_key_iv: "07b87df5cac1d922829f0b843bace8ed"
server_key_version: "encryptionkey@0"
I20260812 02:30:25.764072 22482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:25.790787 22482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:25.795326 22482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:25.795619 22482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:25.796777 22482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:25.798383 22482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:25.798468 22482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:25.798609 22482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:25.798673 22482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:25.836939 22482 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:38651
I20260812 02:30:25.836977 22638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:38651 every 8 connection(s)
I20260812 02:30:25.838430 22482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:30:25.840817 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 22482
I20260812 02:30:25.841027 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:30:25.841868 12323 external_mini_cluster.cc:1428] Setting key d0bf4d98f342a13dd0a8650a4785ba50
Aug 12 02:30:25 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501825, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:25.860176 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:25.841789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:40419 (local address 127.12.8.254:46453)
0812 02:30:25.842987 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:25.842992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:25.843876 (+ 884us) server_negotiation.cc:407] Connection header received
0812 02:30:25.846235 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:25.846241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:25.846349 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:25.846456 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:25.848250 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:25.849045 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:25.850661 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:25.851054 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:25.851164 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:25.854670 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:25.854676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:25.854679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:25.854711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:25.857121 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:25.857878 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:25.857882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:25.857884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:25.858005 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:25.858429 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:25.858433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:25.858435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:25.858699 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:25.858836 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:25.859567 (+ 731us) server_negotiation.cc:299] Negotiation successful
0812 02:30:25.859796 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":953,"threads_started":1}
I20260812 02:30:25.861308 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:25.842444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:40419)
0812 02:30:25.843670 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:25.843722 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:25.845872 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:25.846817 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:25.846828 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:25.847447 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:25.848068 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:25.848087 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:25.849244 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:25.849249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:25.850514 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:25.850527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:25.851308 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:25.851316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:25.851533 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:25.852181 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:25.852210 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:25.854444 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:25.857290 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:25.857299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:25.857316 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:25.857738 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:25.858113 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:25.858118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:25.858122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:25.858311 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:25.858836 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:25.858844 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:25.859176 (+ 332us) client_negotiation.cc:769] Sending connection context
0812 02:30:25.859420 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 02:30:25.859966 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":605,"threads_started":1}
I20260812 02:30:25.863283 22639 heartbeater.cc:347] Connected to a master server at 127.12.8.254:46453
I20260812 02:30:25.863735 22639 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:25.864744 22639 heartbeater.cc:511] Master 127.12.8.254:46453 requested a full tablet report, sending...
I20260812 02:30:25.867642 22406 ts_manager.cc:194] Registered new tserver with Master: fb2f419cf6fd4433b31cd75992e25b01 (127.12.8.193:38651)
I20260812 02:30:25.869517 22406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:40419
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:30:25.918340 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46453
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:26.295271 22648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:26.295612 22648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:26.295744 22648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:26.303922 22648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:26.304087 22648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:26.304260 22648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:30:26.315201 22648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:26.316727 22648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:26.318204 22648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:26.333967 22655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:26.333885 22656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:26.333932 22658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:26.335567 22648 server_base.cc:1034] running on GCE node
I20260812 02:30:26.336433 22648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:26.337821 22648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:26.339119 22648 hybrid_clock.cc:648] HybridClock initialized: now 1786501826339039 us; error 55 us; skew 500 ppm
I20260812 02:30:26.339570 22648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:26 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501826, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:26.343992 22648 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:30:26.346179 22648 webserver.cc:460] Webserver started at http://127.12.8.194:34595/ using document root <none> and password file <none>
I20260812 02:30:26.347112 22648 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:26.347254 22648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:26.347640 22648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:26.351640 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "ce45312e52ca4f49b68bd4fd7b3a391f"
format_stamp: "Formatted at 2026-08-12 02:30:26 on dist-test-slave-vq2q"
server_key: "8d4b304e75d14d1912c77841efeb92b0"
server_key_iv: "b98a565ccba5029aaa8dff1fbdb075af"
server_key_version: "encryptionkey@0"
I20260812 02:30:26.352648 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "ce45312e52ca4f49b68bd4fd7b3a391f"
format_stamp: "Formatted at 2026-08-12 02:30:26 on dist-test-slave-vq2q"
server_key: "8d4b304e75d14d1912c77841efeb92b0"
server_key_iv: "b98a565ccba5029aaa8dff1fbdb075af"
server_key_version: "encryptionkey@0"
I20260812 02:30:26.359500 22648 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260812 02:30:26.363731 22669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:26.365500 22648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:30:26.365705 22648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "ce45312e52ca4f49b68bd4fd7b3a391f"
format_stamp: "Formatted at 2026-08-12 02:30:26 on dist-test-slave-vq2q"
server_key: "8d4b304e75d14d1912c77841efeb92b0"
server_key_iv: "b98a565ccba5029aaa8dff1fbdb075af"
server_key_version: "encryptionkey@0"
I20260812 02:30:26.365969 22648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:26.395656 22648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:26.400851 22648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:26.401269 22648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:26.402508 22648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:26.404171 22648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:26.404259 22648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:26.404392 22648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:26.404446 22648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:26.442090 22648 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:43109
I20260812 02:30:26.442130 22811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:43109 every 8 connection(s)
I20260812 02:30:26.443677 22648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:30:26.446141 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 22648
I20260812 02:30:26.446326 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:30:26.446898 12323 external_mini_cluster.cc:1428] Setting key a7611a645ffb673338ed526bc5c1b89a
Aug 12 02:30:26 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501826, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:26.471292 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:26.446951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:45359 (local address 127.12.8.254:46453)
0812 02:30:26.447545 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:26.447551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:26.448708 (+ 1157us) server_negotiation.cc:407] Connection header received
0812 02:30:26.451621 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:26.451627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:26.451716 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:26.451830 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:26.454385 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:26.455324 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:26.457743 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:26.458293 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:26.458409 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:26.463312 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:26.463319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:26.463324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:26.463371 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:26.466929 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:26.468015 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:26.468020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:26.468025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:26.468221 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:26.468843 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:26.468848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:26.468851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:26.469313 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:26.469471 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:26.470581 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0812 02:30:26.470901 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":273,"threads_started":1}
I20260812 02:30:26.472666 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:26.447627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:45359)
0812 02:30:26.448529 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:26.448575 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:26.451196 (+ 2621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:26.452207 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:26.452223 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:26.453108 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:26.454123 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:26.454153 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:26.455528 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:26.455534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:26.457533 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:26.457560 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:26.458561 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:26.458568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:26.458871 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:26.459760 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:26.459794 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:26.462920 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:26.467187 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:26.467198 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:26.467219 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:26.467836 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:26.468402 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:26.468407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:26.468410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:26.468696 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:26.469492 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:26.469503 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:26.470065 (+ 562us) client_negotiation.cc:769] Sending connection context
0812 02:30:26.470447 (+ 382us) client_negotiation.cc:240] Negotiation successful
0812 02:30:26.471125 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":385,"threads_started":1}
I20260812 02:30:26.474556 22812 heartbeater.cc:347] Connected to a master server at 127.12.8.254:46453
I20260812 02:30:26.474951 22812 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:26.475908 22812 heartbeater.cc:511] Master 127.12.8.254:46453 requested a full tablet report, sending...
I20260812 02:30:26.477900 22406 ts_manager.cc:194] Registered new tserver with Master: ce45312e52ca4f49b68bd4fd7b3a391f (127.12.8.194:43109)
I20260812 02:30:26.478863 22406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:45359
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:30:26.530159 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46453
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:30:26.873611 22639 heartbeater.cc:503] Master 127.12.8.254:46453 was elected leader, sending a full tablet report...
W20260812 02:30:26.949985 22823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:26.950423 22823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:26.950531 22823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:26.958611 22823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:26.958760 22823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:26.958897 22823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:26.970299 22823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:26.972173 22823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:26.973829 22823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:26.991061 22829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:26.990983 22831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:26.991824 22828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:26.991956 22823 server_base.cc:1034] running on GCE node
I20260812 02:30:26.992787 22823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:26.994091 22823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:26.995316 22823 hybrid_clock.cc:648] HybridClock initialized: now 1786501826995261 us; error 50 us; skew 500 ppm
I20260812 02:30:26.995690 22823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:26 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501826, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:26.999969 22823 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:30:27.002352 22823 webserver.cc:460] Webserver started at http://127.12.8.195:46503/ using document root <none> and password file <none>
I20260812 02:30:27.003376 22823 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:27.003538 22823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:27.003952 22823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:27.008158 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "066dacfcb86c40959b8bc1cbe3633721"
format_stamp: "Formatted at 2026-08-12 02:30:26 on dist-test-slave-vq2q"
server_key: "c78567875b6f118634964ae9cd7f86e4"
server_key_iv: "08ba3c8b4c3358445a91afc5472eb2bb"
server_key_version: "encryptionkey@0"
I20260812 02:30:27.009370 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "066dacfcb86c40959b8bc1cbe3633721"
format_stamp: "Formatted at 2026-08-12 02:30:26 on dist-test-slave-vq2q"
server_key: "c78567875b6f118634964ae9cd7f86e4"
server_key_iv: "08ba3c8b4c3358445a91afc5472eb2bb"
server_key_version: "encryptionkey@0"
I20260812 02:30:27.017581 22823 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 02:30:27.022814 22842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:27.024782 22823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:30:27.025010 22823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "066dacfcb86c40959b8bc1cbe3633721"
format_stamp: "Formatted at 2026-08-12 02:30:26 on dist-test-slave-vq2q"
server_key: "c78567875b6f118634964ae9cd7f86e4"
server_key_iv: "08ba3c8b4c3358445a91afc5472eb2bb"
server_key_version: "encryptionkey@0"
I20260812 02:30:27.025311 22823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:27.042493 22823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:27.048563 22823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:27.048931 22823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:27.050247 22823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:27.052000 22823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:27.052093 22823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:27.052238 22823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:27.052307 22823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:27.095994 22823 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:43445
I20260812 02:30:27.096012 22976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:43445 every 8 connection(s)
I20260812 02:30:27.097921 22823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:27.099782 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 22823
I20260812 02:30:27.100003 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:30:27.100595 12323 external_mini_cluster.cc:1428] Setting key edaf4dad71453bac1ebc60c3e755acce
Aug 12 02:30:27 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501826, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:27.123890 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:27.101534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:44425 (local address 127.12.8.254:46453)
0812 02:30:27.102245 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:27.102253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:27.103630 (+ 1377us) server_negotiation.cc:407] Connection header received
0812 02:30:27.106355 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:27.106361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:27.106454 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:27.106583 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:27.108697 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.109678 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.111761 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.112268 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.112385 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:27.117304 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:27.117311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:27.117315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:27.117361 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:27.120331 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:27.121326 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:27.121331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:27.121334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:27.121476 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:27.121940 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:27.121944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:27.121946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:27.122312 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:27.122458 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:27.123333 (+ 875us) server_negotiation.cc:299] Negotiation successful
0812 02:30:27.123550 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":540,"threads_started":1}
I20260812 02:30:27.125216 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:27.102213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:44425)
0812 02:30:27.103395 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:27.103492 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:27.105919 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:27.106929 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:27.106941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:27.107670 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:27.108436 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:27.108459 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.109903 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.109909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:27.111525 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:27.111548 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.112581 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.112589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:27.112861 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:27.113808 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:27.113837 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:27.116983 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:27.120549 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:27.120559 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:27.120580 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:27.121157 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:27.121567 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:27.121570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:27.121572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:27.121821 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:27.122443 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:27.122451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:27.122906 (+ 455us) client_negotiation.cc:769] Sending connection context
0812 02:30:27.123214 (+ 308us) client_negotiation.cc:240] Negotiation successful
0812 02:30:27.123747 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":381,"threads_started":1}
I20260812 02:30:27.127194 22977 heartbeater.cc:347] Connected to a master server at 127.12.8.254:46453
I20260812 02:30:27.127592 22977 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:27.128598 22977 heartbeater.cc:511] Master 127.12.8.254:46453 requested a full tablet report, sending...
I20260812 02:30:27.130642 22405 ts_manager.cc:194] Registered new tserver with Master: 066dacfcb86c40959b8bc1cbe3633721 (127.12.8.195:43445)
I20260812 02:30:27.131704 22405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:44425
I20260812 02:30:27.143914 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab.
Entry for principal oryx/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab.
WARNING: no policy specified for oryx/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab.
Entry for principal oryx/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab.
WARNING: no policy specified for oryx/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab.
Entry for principal oryx/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab.
I20260812 02:30:27.376752 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 22482
Aug 12 02:30:27 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:30:27.394949 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:27.378218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.12.8.193:38651)
0812 02:30:27.378890 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:27.378897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:27.378929 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:30:27.379003 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:27.379010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:27.379237 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:27.379414 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:27.381043 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.382122 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.383689 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.384334 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.384462 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:27.389101 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:27.389113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:27.389127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:27.389161 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:27.392108 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:27.392775 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:27.392782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:27.392784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:27.392915 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:27.393321 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:27.393326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:27.393329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:27.393672 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:27.393989 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:27.394324 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 02:30:27.394591 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":336,"threads_started":1}
I20260812 02:30:27.396458 22634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:27.482348 22812 heartbeater.cc:503] Master 127.12.8.254:46453 was elected leader, sending a full tablet report...
I20260812 02:30:27.484877 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 22482
I20260812 02:30:27.510331 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 22648
Aug 12 02:30:27 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:30:27.531297 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:27.511617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.12.8.194:43109)
0812 02:30:27.512366 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:27.512375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:27.512404 (+ 29us) server_negotiation.cc:407] Connection header received
0812 02:30:27.512477 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:27.512484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:27.512720 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:27.512890 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:27.514578 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.515763 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.517483 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.518123 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.518236 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:27.523554 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:27.523576 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:27.523596 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:27.523661 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:27.527513 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:27.528552 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:27.528562 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:27.528565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:27.528758 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:27.529266 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:27.529270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:27.529272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:27.529745 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:27.530085 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:27.530618 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 02:30:27.530913 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":346,"threads_started":1}
I20260812 02:30:27.533020 22805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:27.626698 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 22648
I20260812 02:30:27.647176 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 22823
Aug 12 02:30:27 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:30:27.665935 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:27.648461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.12.8.195:43445)
0812 02:30:27.649264 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:27.649278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:27.649323 (+ 45us) server_negotiation.cc:407] Connection header received
0812 02:30:27.649419 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:27.649427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:27.649761 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:27.649954 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:27.651397 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.652617 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.654214 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:27.654937 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:27.655104 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:27.659637 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:27.659657 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:27.659678 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:27.659729 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:27.663041 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:27.663746 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:27.663754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:27.663757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:27.663906 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:27.664218 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:27.664222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:27.664224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:27.664674 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:27.664959 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:27.665151 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 02:30:27.665490 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":348,"threads_started":1}
I20260812 02:30:27.667640 22972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:27.761546 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 22823
I20260812 02:30:27.799041 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 22369
I20260812 02:30:27.800669 22445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:27.930804 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 22369
I20260812 02:30:27.963639 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46453
--webserver_interface=127.12.8.254
--webserver_port=32865
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:46453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:28.395267 23000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:28.395607 23000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:28.395706 23000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:28.404398 23000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:28.404552 23000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:28.404603 23000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:28.404656 23000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:28.404707 23000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:28.415534 23000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:46453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=32865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:28.417104 23000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:28.418576 23000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:28.432376 23006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:28.432525 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:28.432622 23007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:28.434103 23000 server_base.cc:1034] running on GCE node
I20260812 02:30:28.435141 23000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:28.436825 23000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:28.438062 23000 hybrid_clock.cc:648] HybridClock initialized: now 1786501828437967 us; error 96 us; skew 500 ppm
I20260812 02:30:28.438436 23000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:28 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501828, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:28.441918 23000 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:30:28.443650 23000 webserver.cc:460] Webserver started at http://127.12.8.254:32865/ using document root <none> and password file <none>
I20260812 02:30:28.444442 23000 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:28.444574 23000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:28.451521 23000 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:30:28.455672 23021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:28.457722 23000 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260812 02:30:28.457950 23000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "d294b912188c4e97a2550dfb62afce6e"
format_stamp: "Formatted at 2026-08-12 02:30:24 on dist-test-slave-vq2q"
server_key: "ed329ab789dff3f6fa91d22cd4cbfc2d"
server_key_iv: "f29cf8725bb7dfbe8317783840838482"
server_key_version: "encryptionkey@0"
I20260812 02:30:28.459088 23000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:28.485347 23000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:28.490374 23000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:28.490644 23000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:28.512491 23000 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:46453
I20260812 02:30:28.512554 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:46453 every 8 connection(s)
I20260812 02:30:28.514343 23000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:30:28.521956 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 23000
I20260812 02:30:28.523133 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:38651
--local_ip_for_outbound_sockets=127.12.8.193
--tserver_master_addrs=127.12.8.254:46453
--webserver_port=39371
--webserver_interface=127.12.8.193
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.8.193
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:30:28.524185 23083 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:30:28.530385 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Bootstrap starting.
I20260812 02:30:28.558676 23083 log.cc:824] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:28.572028 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:30:28.572849 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Bootstrap complete.
I20260812 02:30:28.586284 23083 raft_consensus.cc:348] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } }
I20260812 02:30:28.587384 23083 raft_consensus.cc:729] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d294b912188c4e97a2550dfb62afce6e, State: Initialized, Role: FOLLOWER
I20260812 02:30:28.588251 23083 consensus_queue.cc:260] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } }
I20260812 02:30:28.588528 23083 raft_consensus.cc:388] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:30:28.588670 23083 raft_consensus.cc:482] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:30:28.588858 23083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:30:28.594333 23083 raft_consensus.cc:504] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } }
I20260812 02:30:28.595060 23083 leader_election.cc:302] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d294b912188c4e97a2550dfb62afce6e; no voters:
I20260812 02:30:28.595831 23083 leader_election.cc:288] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:30:28.596005 23087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:30:28.597440 23087 raft_consensus.cc:686] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [term 2 LEADER]: Becoming Leader. State: Replica: d294b912188c4e97a2550dfb62afce6e, State: Running, Role: LEADER
I20260812 02:30:28.598240 23087 consensus_queue.cc:237] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } }
I20260812 02:30:28.598614 23083 sys_catalog.cc:564] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:28.601420 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d294b912188c4e97a2550dfb62afce6e. Latest consensus state: current_term: 2 leader_uuid: "d294b912188c4e97a2550dfb62afce6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } } }
I20260812 02:30:28.602101 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:28.603214 23089 sys_catalog.cc:455] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d294b912188c4e97a2550dfb62afce6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d294b912188c4e97a2550dfb62afce6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46453 } } }
I20260812 02:30:28.606247 23089 sys_catalog.cc:458] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:28.610611 23096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:30:28.619124 23096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:30:28.621054 23096 catalog_manager.cc:1259] Loaded cluster ID: 05bad871e85b46798ab8bfce84990528
I20260812 02:30:28.621829 23096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:30:28.628976 23096 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:30:28.631359 23096 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d294b912188c4e97a2550dfb62afce6e: Loaded TSK: 0
I20260812 02:30:28.632907 23096 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:30:28.993965 23085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:28.994382 23085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:28.994498 23085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:29.003158 23085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:29.003321 23085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:29.003455 23085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:30:29.015794 23085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.193.keytab
--principal=oryx/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:38651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=39371
--tserver_master_addrs=127.12.8.254:46453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:29.017733 23085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:29.019512 23085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:29.037158 23118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:29.037271 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:29.037158 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:29.043179 23085 server_base.cc:1034] running on GCE node
I20260812 02:30:29.044070 23085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:29.045476 23085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:29.046730 23085 hybrid_clock.cc:648] HybridClock initialized: now 1786501829046660 us; error 56 us; skew 500 ppm
I20260812 02:30:29.047130 23085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501829, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:29.051147 23085 init.cc:377] Logged in from keytab as oryx/127.12.8.193@KRBTEST.COM (short username oryx)
I20260812 02:30:29.053162 23085 webserver.cc:460] Webserver started at http://127.12.8.193:39371/ using document root <none> and password file <none>
I20260812 02:30:29.054111 23085 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:29.054246 23085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:29.061321 23085 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20260812 02:30:29.065343 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:29.067461 23085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:30:29.067741 23085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "fb2f419cf6fd4433b31cd75992e25b01"
format_stamp: "Formatted at 2026-08-12 02:30:25 on dist-test-slave-vq2q"
server_key: "fa9567b2d9688b17fa824f206daf907a"
server_key_iv: "07b87df5cac1d922829f0b843bace8ed"
server_key_version: "encryptionkey@0"
I20260812 02:30:29.069054 23085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:29.095528 23085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:29.101387 23085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:29.101747 23085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:29.103143 23085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:29.104952 23085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:29.105068 23085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:29.105244 23085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:29.105331 23085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:29.146183 23085 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:38651
I20260812 02:30:29.146245 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:38651 every 8 connection(s)
I20260812 02:30:29.148000 23085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:30:29.157888 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 23085
I20260812 02:30:29.158710 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:43109
--local_ip_for_outbound_sockets=127.12.8.194
--tserver_master_addrs=127.12.8.254:46453
--webserver_port=34595
--webserver_interface=127.12.8.194
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.8.194
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:30:29.168330 23273 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:29.151553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:59541 (local address 127.12.8.254:46453)
0812 02:30:29.152284 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:29.152303 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:29.152944 (+ 641us) server_negotiation.cc:407] Connection header received
0812 02:30:29.155392 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:29.155404 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:29.155764 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:29.156093 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:29.157837 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.159164 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.160785 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.161491 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.161675 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.167268 (+ 5593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:59541: BlockingRecv error: recv got EOF from 127.12.8.193:59541 (error 108)
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":278,"threads_started":1}
I20260812 02:30:29.168345 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:29.151827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:59541)
0812 02:30:29.152746 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:29.152793 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:29.154812 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:29.156309 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:29.156320 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:29.156962 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:29.157627 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.157647 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.159344 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.159348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.160621 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.160635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.161755 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.161764 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.162022 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.162704 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:29.162734 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:29.166923 (+ 4189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":390,"threads_started":1}
W20260812 02:30:29.169538 23271 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:46453 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.8.254:46453: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:29.177947 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:29.170152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:57493)
0812 02:30:29.170342 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:29.170360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:29.170448 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:29.170946 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:29.170950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:29.171296 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:29.171439 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.171448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.172453 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.172457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.173467 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.173480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.174145 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.174151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.174340 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.174887 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:29.174905 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:29.177671 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 02:30:29.178135 23273 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:29.170470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:57493 (local address 127.12.8.254:46453)
0812 02:30:29.170670 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:29.170675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:29.170690 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:30:29.170745 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:29.170749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:29.170807 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:29.170890 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:29.171599 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.172301 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.173602 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.173996 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.174139 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.177903 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:57493: BlockingRecv error: recv got EOF from 127.12.8.193:57493 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:29.188457 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:29.178632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:39943)
0812 02:30:29.179268 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:29.179285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:29.179406 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:29.179945 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:29.179949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:29.180347 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:29.180564 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.180573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.182034 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.182038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.182998 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.183009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.184083 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.184088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.184224 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.184901 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:29.184920 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:29.188211 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":283,"threads_started":1}
W20260812 02:30:29.188833 23271 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:30:29.188927 23273 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:29.178850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:39943 (local address 127.12.8.254:46453)
0812 02:30:29.179093 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:29.179099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:29.179390 (+ 291us) server_negotiation.cc:407] Connection header received
0812 02:30:29.179555 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:29.179564 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:29.179702 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:29.179893 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:29.180745 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.181862 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.183191 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.183840 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.184030 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.188645 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:39943: BlockingRecv error: recv got EOF from 127.12.8.193:39943 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260812 02:30:29.581925 23276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:29.582255 23276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:29.582360 23276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:29.589946 23276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:29.590091 23276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:29.590268 23276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:30:29.600716 23276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.194.keytab
--principal=oryx/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:43109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=34595
--tserver_master_addrs=127.12.8.254:46453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:29.602183 23276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:29.603544 23276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:29.618377 23282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:29.618345 23286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:29.618666 23284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:29.619963 23276 server_base.cc:1034] running on GCE node
I20260812 02:30:29.620657 23276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:29.621812 23276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:29.623034 23276 hybrid_clock.cc:648] HybridClock initialized: now 1786501829622966 us; error 70 us; skew 500 ppm
I20260812 02:30:29.623364 23276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501829, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:29.626806 23276 init.cc:377] Logged in from keytab as oryx/127.12.8.194@KRBTEST.COM (short username oryx)
I20260812 02:30:29.628563 23276 webserver.cc:460] Webserver started at http://127.12.8.194:34595/ using document root <none> and password file <none>
I20260812 02:30:29.629375 23276 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:29.629510 23276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:29.636055 23276 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 02:30:29.639513 23293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:29.641521 23276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:30:29.641736 23276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "ce45312e52ca4f49b68bd4fd7b3a391f"
format_stamp: "Formatted at 2026-08-12 02:30:26 on dist-test-slave-vq2q"
server_key: "8d4b304e75d14d1912c77841efeb92b0"
server_key_iv: "b98a565ccba5029aaa8dff1fbdb075af"
server_key_version: "encryptionkey@0"
I20260812 02:30:29.642890 23276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:29.696010 23276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:29.700739 23276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:29.701020 23276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:29.702247 23276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:29.703795 23276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:29.703896 23276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:29.704011 23276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:29.704082 23276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:29.743407 23276 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:43109
I20260812 02:30:29.743532 23431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:43109 every 8 connection(s)
I20260812 02:30:29.745062 23276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:30:29.754367 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 23276
I20260812 02:30:29.755082 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:43445
--local_ip_for_outbound_sockets=127.12.8.195
--tserver_master_addrs=127.12.8.254:46453
--webserver_port=46503
--webserver_interface=127.12.8.195
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.8.195
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:30:29.767459 23434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:29.748889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:49353 (local address 127.12.8.254:46453)
0812 02:30:29.749600 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:29.749607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:29.750842 (+ 1235us) server_negotiation.cc:407] Connection header received
0812 02:30:29.753938 (+ 3096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:29.753945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:29.754223 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:29.754547 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:29.757069 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.757969 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.759915 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.760368 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.760503 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.767103 (+ 6600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:49353: BlockingRecv error: recv got EOF from 127.12.8.194:49353 (error 108)
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":437,"threads_started":1}
I20260812 02:30:29.768394 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:29.749428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:49353)
0812 02:30:29.750601 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:29.750671 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:29.753451 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:29.754981 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:29.754994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:29.755820 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:29.756652 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.756679 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.758190 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.758196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.759719 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.759739 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.760791 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.760805 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.761132 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.762156 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:29.762196 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:29.766799 (+ 4603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":379,"threads_started":1}
W20260812 02:30:29.769484 23432 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:46453 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.8.254:46453: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:29.781745 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:29.770249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:42219)
0812 02:30:29.770502 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:29.770526 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:29.770725 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:29.771089 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:29.771094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:29.771527 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:29.771719 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.771731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.772856 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.772861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.774282 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.774304 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.775741 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.775749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.775976 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.776740 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:29.776767 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:29.781352 (+ 4585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260812 02:30:29.782366 23434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:29.770454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:42219 (local address 127.12.8.254:46453)
0812 02:30:29.770651 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:29.770658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:29.770681 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:30:29.770866 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:29.770872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:29.770943 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:29.771034 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:29.771885 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.772688 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.774526 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.775266 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.775446 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.782012 (+ 6566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:42219: BlockingRecv error: recv got EOF from 127.12.8.194:42219 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:29 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:29.793572 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:29.783273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:50555)
0812 02:30:29.783521 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:29.783548 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:29.783715 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:29.784122 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:29.784127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:29.784570 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:29.784845 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.784859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.786217 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.786223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.787326 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:29.787344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.788360 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.788368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:29.788590 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.789406 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:29.789429 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:29.793119 (+ 3690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260812 02:30:29.793910 23434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:29.783427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:50555 (local address 127.12.8.254:46453)
0812 02:30:29.783664 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:29.783671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:29.783695 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:30:29.783863 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:29.783870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:29.783956 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:29.784064 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:29.785044 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.786025 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.787582 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:29.788137 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:29.788306 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:29.793564 (+ 5258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:50555: BlockingRecv error: recv got EOF from 127.12.8.194:50555 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260812 02:30:29.794260 23432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:30:30.119239 23436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:30.119549 23436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:30.119647 23436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:30.127064 23436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:30.127197 23436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:30.127314 23436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:30.137373 23436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/krb5kdc/oryx_127.12.8.195.keytab
--principal=oryx/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:43445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=46503
--tserver_master_addrs=127.12.8.254:46453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:30.138729 23436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:30.140023 23436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:30.154243 23443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:30.154412 23448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:30.154587 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:30.155421 23436 server_base.cc:1034] running on GCE node
I20260812 02:30:30.156262 23436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:30.157469 23436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:30.158712 23436 hybrid_clock.cc:648] HybridClock initialized: now 1786501830158644 us; error 48 us; skew 500 ppm
I20260812 02:30:30.159122 23436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501830, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:30.165894 23436 init.cc:377] Logged in from keytab as oryx/127.12.8.195@KRBTEST.COM (short username oryx)
I20260812 02:30:30.167642 23436 webserver.cc:460] Webserver started at http://127.12.8.195:46503/ using document root <none> and password file <none>
I20260812 02:30:30.168381 23436 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:30.168517 23436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:30.174714 23436 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:30:30.178023 23457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:30.179587 23436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:30:30.179811 23436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "066dacfcb86c40959b8bc1cbe3633721"
format_stamp: "Formatted at 2026-08-12 02:30:26 on dist-test-slave-vq2q"
server_key: "c78567875b6f118634964ae9cd7f86e4"
server_key_iv: "08ba3c8b4c3358445a91afc5472eb2bb"
server_key_version: "encryptionkey@0"
I20260812 02:30:30.180830 23436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:30.198540 23436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:30.199667 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:30.189702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:34375)
0812 02:30:30.190438 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:30.190458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:30.190670 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:30.191058 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:30.191065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:30.191610 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:30.191831 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.191843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.192900 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.192905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.193951 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.193967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.194791 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.194797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.194985 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.195540 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:30.195559 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:30.199249 (+ 3690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":462,"threads_started":1}
W20260812 02:30:30.199841 23434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:30.189867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:34375 (local address 127.12.8.254:46453)
0812 02:30:30.190123 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:30.190130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:30.190582 (+ 452us) server_negotiation.cc:407] Connection header received
0812 02:30:30.190817 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:30.190821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:30.190907 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:30.190999 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:30.191998 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.192757 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.194122 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.194592 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.194710 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.199578 (+ 4868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:34375: BlockingRecv error: recv got EOF from 127.12.8.193:34375 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 02:30:30.204071 23436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:30.204339 23436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:30.205519 23436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:30.207063 23436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:30.207165 23436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:30.207283 23436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:30.207360 23436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:30.242520 23436 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:43445
I20260812 02:30:30.242659 23607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:43445 every 8 connection(s)
I20260812 02:30:30.244093 23436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:30.252092 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 23436
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:30:30.267036 23434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:30.247673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:53329 (local address 127.12.8.254:46453)
0812 02:30:30.247876 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:30.247882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:30.249684 (+ 1802us) server_negotiation.cc:407] Connection header received
0812 02:30:30.252423 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:30.252429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:30.252524 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:30.252624 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:30.255267 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.256496 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.259078 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.259766 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.259883 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.266716 (+ 6833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:53329: BlockingRecv error: recv got EOF from 127.12.8.195:53329 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:30:30.267838 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:30.248114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:53329)
0812 02:30:30.249219 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:30.249305 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:30.251853 (+ 2548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:30.253002 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:30.253016 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:30.253943 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:30.255005 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.255045 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.256747 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.256752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.258901 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.258918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.260127 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.260138 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.260550 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.261555 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:30.261588 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:30.266492 (+ 4904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":389,"threads_started":1}
W20260812 02:30:30.269526 23608 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:46453 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.8.254:46453: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
I20260812 02:30:30.272337 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:30.253339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.12.8.254:46453)
0812 02:30:30.254089 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:30.254096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:30.254433 (+ 337us) server_negotiation.cc:407] Connection header received
0812 02:30:30.254615 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:30.254621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:30.254728 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:30.254894 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:30.256933 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.258224 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.259808 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.260518 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.260702 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.263804 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:30.263820 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:30.263847 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:30.263907 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:30.267865 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:30.268692 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:30.268700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:30.268709 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:30.268951 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:30.269473 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:30.269477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:30.269480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:30.270227 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:30.270581 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:30.271213 (+ 632us) server_negotiation.cc:299] Negotiation successful
0812 02:30:30.271811 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":374,"threads_started":1}
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:30.281533 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:30.270358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:59293)
0812 02:30:30.270684 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:30.270705 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:30.270829 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:30.271227 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:30.271232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:30.271629 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:30.271793 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.271802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.274116 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.274123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.275546 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.275567 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.276634 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.276641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.276842 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.277658 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:30.277683 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:30.281115 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189}
W20260812 02:30:30.281724 23434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:30.270536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:59293 (local address 127.12.8.254:46453)
0812 02:30:30.270814 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:30.270818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:30.270835 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:30:30.270964 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:30.270970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:30.271055 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:30.271171 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:30.271956 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.273887 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.275791 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.276442 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.276578 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.281481 (+ 4903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:59293: BlockingRecv error: recv got EOF from 127.12.8.195:59293 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:30.291544 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:30.282448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:46539)
0812 02:30:30.282660 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:30.282682 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:30.282822 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:30.283202 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:30.283207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:30.283548 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:30.283732 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.283741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.284917 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.284921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.285998 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.286012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.286730 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.286736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.286947 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.287582 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:30.287629 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:30.291288 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 02:30:30.291872 23434 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:30.282628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:46539 (local address 127.12.8.254:46453)
0812 02:30:30.282816 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:30.282822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:30.282843 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:30:30.282955 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:30.282961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:30.283031 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:30.283118 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:30.283876 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.284776 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.286139 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.286574 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.286696 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.291554 (+ 4858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:46539: BlockingRecv error: recv got EOF from 127.12.8.195:46539 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 02:30:30.291887 23608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:30 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:30.805568 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:30.794983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:55769)
0812 02:30:30.795623 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:30.795643 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:30.795809 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:30.796242 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:30.796247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:30.796788 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:30.797016 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.797026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.798309 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.798313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.799469 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:30.799484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.800389 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.800394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:30.800521 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.801268 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:30.801293 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:30.805210 (+ 3917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":290,"threads_started":1}
W20260812 02:30:30.805826 23615 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:30.795120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:55769 (local address 127.12.8.254:46453)
0812 02:30:30.795910 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:30.795915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:30.795935 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:30:30.795989 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:30.795993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:30.796074 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:30.796180 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:30.797170 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.798135 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.799650 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:30.800209 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:30.800327 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:30.805541 (+ 5214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:55769: BlockingRecv error: recv got EOF from 127.12.8.194:55769 (error 108)
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":559,"threads_started":1}
Aug 12 02:30:31 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:31 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:31.212775 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:31.202812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:39025)
0812 02:30:31.203501 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:31.203522 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:31.203700 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:31.204057 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:31.204062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:31.204646 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:31.204887 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:31.204910 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.206219 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.206225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:31.207434 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:31.207448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.208281 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.208287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:31.208449 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:31.209118 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:31.209139 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:31.212421 (+ 3282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":360,"threads_started":1}
W20260812 02:30:31.213166 23615 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:31.202909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:39025 (local address 127.12.8.254:46453)
0812 02:30:31.203185 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:31.203192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:31.203638 (+ 446us) server_negotiation.cc:407] Connection header received
0812 02:30:31.203833 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:31.203838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:31.203907 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:31.203992 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:31.205087 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.206017 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.207576 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.208111 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.208237 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:31.212868 (+ 4631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:39025: BlockingRecv error: recv got EOF from 127.12.8.193:39025 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 12 02:30:31 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:31 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:31.303503 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:31.292614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:40635)
0812 02:30:31.293509 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:31.293532 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:31.293723 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:31.294274 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:31.294279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:31.294998 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:31.295389 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:31.295401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.296590 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.296595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:31.297789 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:31.297806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.298734 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.298742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:31.298965 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:31.299793 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:31.299814 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:31.303197 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":610,"threads_started":1}
W20260812 02:30:31.303835 23615 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:31.292673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:40635 (local address 127.12.8.254:46453)
0812 02:30:31.292899 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:31.292908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:31.293651 (+ 743us) server_negotiation.cc:407] Connection header received
0812 02:30:31.293923 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:31.293930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:31.294038 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:31.294168 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:31.295618 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.296425 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.297971 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.298451 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.298617 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:31.303512 (+ 4895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:40635: BlockingRecv error: recv got EOF from 127.12.8.195:40635 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 02:30:31 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:31 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:31.817373 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:31.806981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:57935)
0812 02:30:31.807638 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:31.807659 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:31.807828 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:31.808215 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:31.808218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:31.808746 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:31.808954 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:31.808965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.810123 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.810128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:31.811261 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:31.811278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.812236 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.812242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:31.812373 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:31.813135 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:31.813155 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:31.816906 (+ 3751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":352,"threads_started":1}
W20260812 02:30:31.817555 23622 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:31.807103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:57935 (local address 127.12.8.254:46453)
0812 02:30:31.807585 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:31.807590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:31.807753 (+ 163us) server_negotiation.cc:407] Connection header received
0812 02:30:31.807939 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:31.807943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:31.808018 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:31.808138 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:31.809108 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.809921 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.811414 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:31.812049 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:31.812154 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:31.817252 (+ 5098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:57935: BlockingRecv error: recv got EOF from 127.12.8.194:57935 (error 108)
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
Aug 12 02:30:32 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:32 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:32.222663 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:32.214011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:35065)
0812 02:30:32.214729 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:32.214748 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:32.214918 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:32.215274 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:32.215279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:32.215676 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:32.215846 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:32.215856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.216882 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.216887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:32.218063 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:32.218078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.218755 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.218761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:32.218920 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:32.219515 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:32.219532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:32.222313 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":345,"threads_started":1}
W20260812 02:30:32.222810 23622 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:32.214281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:35065 (local address 127.12.8.254:46453)
0812 02:30:32.214544 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:32.214552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:32.214836 (+ 284us) server_negotiation.cc:407] Connection header received
0812 02:30:32.215060 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:32.215066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:32.215135 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:32.215220 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:32.215998 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.216737 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.218202 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.218602 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.218731 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:32.222566 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:35065: BlockingRecv error: recv got EOF from 127.12.8.193:35065 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 02:30:32 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:32 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:32.314203 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:32.304932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:52065)
0812 02:30:32.305640 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:32.305665 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:32.305876 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:32.306327 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:32.306333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:32.306842 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:32.307016 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:32.307026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.308106 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.308113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:32.309081 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:32.309092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.309841 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.309846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:32.310015 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:32.310656 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:32.310676 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:32.313827 (+ 3151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":353,"threads_started":1}
W20260812 02:30:32.314409 23622 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:32.305353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:52065 (local address 127.12.8.254:46453)
0812 02:30:32.305627 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:32.305634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:32.305789 (+ 155us) server_negotiation.cc:407] Connection header received
0812 02:30:32.306016 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:32.306022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:32.306146 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:32.306243 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:32.307174 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.307960 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.309248 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.309681 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.309792 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:32.314119 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:52065: BlockingRecv error: recv got EOF from 127.12.8.195:52065 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 12 02:30:32 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:32 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:32.828960 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:32.818765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:56601)
0812 02:30:32.819471 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:32.819498 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:32.819708 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:32.820141 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:32.820146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:32.820599 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:32.820816 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:32.820829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.822152 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.822156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:32.823162 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:32.823176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.824018 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.824023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:32.824232 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:32.824999 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:32.825025 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:32.828609 (+ 3584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":346,"threads_started":1}
W20260812 02:30:32.829303 23627 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:32.818887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:56601 (local address 127.12.8.254:46453)
0812 02:30:32.819761 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:32.819769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:32.819791 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:30:32.819863 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:32.819867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:32.819960 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:32.820073 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:32.821003 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.821985 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.823328 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:32.823842 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:32.823969 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:32.828952 (+ 4983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:56601: BlockingRecv error: recv got EOF from 127.12.8.194:56601 (error 108)
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":535,"threads_started":1}
Aug 12 02:30:33 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:33 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:33.234010 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:33.224062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:40325)
0812 02:30:33.224698 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:33.224721 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:33.224894 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:33.225482 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:33.225488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:33.226044 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:33.226244 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:33.226259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.227422 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.227427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:33.228527 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:33.228542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.229335 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.229342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:33.229553 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:33.230284 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:33.230305 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:33.233715 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":347,"threads_started":1}
W20260812 02:30:33.234256 23627 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:33.224073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:40325 (local address 127.12.8.254:46453)
0812 02:30:33.224300 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:33.224310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:33.224826 (+ 516us) server_negotiation.cc:407] Connection header received
0812 02:30:33.225027 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:33.225034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:33.225136 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:33.225284 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:33.226408 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.227251 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.228683 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.229141 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.229314 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:33.233985 (+ 4671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:40325: BlockingRecv error: recv got EOF from 127.12.8.193:40325 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 02:30:33 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:33 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:33.326293 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:33.315476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:44843)
0812 02:30:33.316031 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:33.316058 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:33.316220 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:33.316623 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:33.316630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:33.317168 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:33.317465 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:33.317480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.318769 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.318775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:33.320327 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:33.320347 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.321132 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.321138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:33.321392 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:33.322163 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:33.322190 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:33.325894 (+ 3704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":495,"threads_started":1}
W20260812 02:30:33.326700 23627 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:33.315699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:44843 (local address 127.12.8.254:46453)
0812 02:30:33.315973 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:33.315980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:33.316235 (+ 255us) server_negotiation.cc:407] Connection header received
0812 02:30:33.316336 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:33.316341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:33.316446 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:33.316556 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:33.317718 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.318598 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.320501 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.320966 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.321107 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:33.326210 (+ 5103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:44843: BlockingRecv error: recv got EOF from 127.12.8.195:44843 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 12 02:30:33 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:33 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:33.839982 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:33.830048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:48381)
0812 02:30:33.830749 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:33.830774 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:33.830935 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:33.831331 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:33.831335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:33.831795 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:33.832004 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:33.832016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.833133 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.833136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:33.834237 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:33.834251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.835103 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.835134 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:33.835285 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:33.836014 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:33.836034 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:33.839711 (+ 3677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":356,"threads_started":1}
W20260812 02:30:33.840420 23631 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:33.830173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:48381 (local address 127.12.8.254:46453)
0812 02:30:33.830855 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:33.830862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:33.830888 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:30:33.831061 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:33.831065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:33.831145 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:33.831257 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:33.832167 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.832990 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.834426 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:33.834950 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:33.835071 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:33.840062 (+ 4991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:48381: BlockingRecv error: recv got EOF from 127.12.8.194:48381 (error 108)
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":379,"threads_started":1}
Aug 12 02:30:34 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:34 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:34.244545 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:34.235159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:49713)
0812 02:30:34.235899 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:34.235921 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:34.236096 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:34.236421 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:34.236426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:34.236933 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:34.237127 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:34.237138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.238191 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.238198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:34.239132 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:34.239144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.239806 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.239812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:34.239991 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:34.240604 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:34.240623 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:34.244130 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":341,"threads_started":1}
W20260812 02:30:34.244690 23631 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:34.235297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:49713 (local address 127.12.8.254:46453)
0812 02:30:34.235531 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:34.235539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:34.236024 (+ 485us) server_negotiation.cc:407] Connection header received
0812 02:30:34.236215 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:34.236219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:34.236284 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:34.236375 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:34.237300 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.238051 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.239274 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.239658 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.239766 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:34.244437 (+ 4671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:49713: BlockingRecv error: recv got EOF from 127.12.8.193:49713 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 02:30:34 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:34 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:34.337651 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:34.327557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:38065)
0812 02:30:34.328237 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:34.328257 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:34.328434 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:34.328831 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:34.328836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:34.329361 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:34.329561 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:34.329571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.330658 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.330663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:34.331729 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:34.331741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.332578 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.332584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:34.332787 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:34.333560 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:34.333578 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:34.337339 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":382,"threads_started":1}
W20260812 02:30:34.338193 23631 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:34.327714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:38065 (local address 127.12.8.254:46453)
0812 02:30:34.328009 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:34.328052 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:34.328356 (+ 304us) server_negotiation.cc:407] Connection header received
0812 02:30:34.328570 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:34.328576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:34.328685 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:34.328779 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:34.329713 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.330522 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.331878 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.332406 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.332541 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:34.337678 (+ 5137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:38065: BlockingRecv error: recv got EOF from 127.12.8.195:38065 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 12 02:30:34 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:34 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:34.850989 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:34.841304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:55691)
0812 02:30:34.841904 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:34.841930 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:34.842106 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:34.842598 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:34.842602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:34.843096 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:34.843309 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:34.843321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.844376 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.844381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:34.845481 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:34.845496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.846266 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.846271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:34.846459 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:34.847185 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:34.847206 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:34.850629 (+ 3423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":286,"threads_started":1}
W20260812 02:30:34.851321 23638 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:34.841540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:55691 (local address 127.12.8.254:46453)
0812 02:30:34.842117 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:34.842122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:34.842142 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:30:34.842257 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:34.842262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:34.842353 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:34.842465 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:34.843443 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.844218 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.845633 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:34.846102 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:34.846209 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:34.850901 (+ 4692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:55691: BlockingRecv error: recv got EOF from 127.12.8.194:55691 (error 108)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":333,"threads_started":1}
Aug 12 02:30:35 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:35 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:35.260018 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:35.245922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:40365)
0812 02:30:35.246655 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:35.246697 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:35.246848 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:35.247285 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:35.247293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:35.247828 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:35.248054 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:35.248071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.249312 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.249317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:35.250407 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:35.250429 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.251225 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.251230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:35.251387 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:35.255453 (+ 4066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:35.255485 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:35.259686 (+ 4201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":414,"threads_started":1}
W20260812 02:30:35.260252 23638 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:35.246020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:40365 (local address 127.12.8.254:46453)
0812 02:30:35.246208 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:35.246213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:35.246799 (+ 586us) server_negotiation.cc:407] Connection header received
0812 02:30:35.246961 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:35.246965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:35.247038 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:35.247145 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:35.248236 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.249045 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.250593 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.251003 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.251101 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:35.260017 (+ 8916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:40365: BlockingRecv error: recv got EOF from 127.12.8.193:40365 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 02:30:35 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:35 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:35.348258 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:35.339005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:55893)
0812 02:30:35.339755 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:35.339777 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:35.339943 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:35.340307 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:35.340311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:35.340781 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:35.340961 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:35.340971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.342182 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.342186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:35.343261 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:35.343275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.344034 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.344040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:35.344202 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:35.344898 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:35.344917 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:35.347892 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":390,"threads_started":1}
W20260812 02:30:35.348409 23638 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:35.339179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:55893 (local address 127.12.8.254:46453)
0812 02:30:35.339477 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:35.339492 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:35.339887 (+ 395us) server_negotiation.cc:407] Connection header received
0812 02:30:35.340068 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:35.340072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:35.340151 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:35.340241 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:35.341120 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.342016 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.343408 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.343880 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.343992 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:35.348176 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:55893: BlockingRecv error: recv got EOF from 127.12.8.195:55893 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 12 02:30:35 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:35 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:35.863987 23642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:35.852256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:40819)
0812 02:30:35.852894 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:35.852915 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:35.853071 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:35.853613 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:35.853620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:35.854337 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:35.854594 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:35.854610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.856177 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.856183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:35.857782 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:35.857803 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.858805 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.858812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:35.859008 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:35.859846 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:35.859876 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:35.863540 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":376,"threads_started":1}
W20260812 02:30:35.864317 23641 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:35.852272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:40819 (local address 127.12.8.254:46453)
0812 02:30:35.852813 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:35.852819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:35.853042 (+ 223us) server_negotiation.cc:407] Connection header received
0812 02:30:35.853285 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:35.853290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:35.853398 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:35.853549 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:35.854771 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.855983 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.857953 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:35.858610 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:35.859035 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:35.863944 (+ 4909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:40819: BlockingRecv error: recv got EOF from 127.12.8.194:40819 (error 108)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
Aug 12 02:30:36 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:36 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:36.271807 23643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:36.261408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:44723)
0812 02:30:36.262148 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:36.262170 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:36.262357 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:36.262832 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:36.262836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:36.263470 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:36.263689 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:36.263700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.264940 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.264945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:36.266089 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:36.266107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.266962 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.266968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:36.267170 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:36.267888 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:36.267909 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:36.271477 (+ 3568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":297,"threads_started":1}
W20260812 02:30:36.272073 23641 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:36.261463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:44723 (local address 127.12.8.254:46453)
0812 02:30:36.261682 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:36.261688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:36.262287 (+ 599us) server_negotiation.cc:407] Connection header received
0812 02:30:36.262496 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:36.262502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:36.262598 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:36.262705 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:36.263856 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.264744 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.266270 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.266763 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.266874 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:36.271809 (+ 4935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:44723: BlockingRecv error: recv got EOF from 127.12.8.193:44723 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 02:30:36 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:36 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:36.359550 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:36.349706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:34423)
0812 02:30:36.350407 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:36.350426 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:36.350610 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:36.351003 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:36.351009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:36.351659 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:36.351942 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:36.351956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.353137 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.353141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:36.354172 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:36.354189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.354953 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.354960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:36.355277 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:36.355987 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:36.356054 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:36.359274 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":432,"threads_started":1}
W20260812 02:30:36.360103 23641 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:36.349907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:34423 (local address 127.12.8.254:46453)
0812 02:30:36.350188 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:36.350196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:36.350548 (+ 352us) server_negotiation.cc:407] Connection header received
0812 02:30:36.350749 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:36.350753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:36.350847 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:36.350944 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:36.352117 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.352980 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.354338 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.354779 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.354899 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:36.359622 (+ 4723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:34423: BlockingRecv error: recv got EOF from 127.12.8.195:34423 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 12 02:30:36 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:36 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:36.875241 23645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:36.865380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:40239)
0812 02:30:36.866051 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:36.866071 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:36.866190 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:36.866704 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:36.866707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:36.867183 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:36.867381 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:36.867391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.868808 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.868812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:36.869990 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:36.870004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.870981 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.870988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:36.871126 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:36.871820 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:36.871841 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:36.874952 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":281,"threads_started":1}
W20260812 02:30:36.875686 23646 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:36.865480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:40239 (local address 127.12.8.254:46453)
0812 02:30:36.866135 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:36.866140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:36.866167 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:30:36.866392 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:36.866398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:36.866503 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:36.866648 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:36.867584 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.868621 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.870129 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:36.870755 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:36.870906 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:36.875322 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:40239: BlockingRecv error: recv got EOF from 127.12.8.194:40239 (error 108)
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":331,"threads_started":1}
Aug 12 02:30:37 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:37 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:37.284488 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:37.273288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:52977)
0812 02:30:37.273968 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:37.273988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:37.274180 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:37.274592 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:37.274596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:37.275158 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:37.275397 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:37.275409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.276556 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.276562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:37.277857 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:37.277879 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.278772 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.278778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:37.278933 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:37.279838 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:37.279862 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:37.284142 (+ 4280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":389,"threads_started":1}
W20260812 02:30:37.284690 23646 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:37.273322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:52977 (local address 127.12.8.254:46453)
0812 02:30:37.273509 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:37.273513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:37.274086 (+ 573us) server_negotiation.cc:407] Connection header received
0812 02:30:37.274331 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:37.274336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:37.274433 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:37.274543 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:37.275586 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.276376 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.278078 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.278527 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.278654 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:37.284450 (+ 5796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:52977: BlockingRecv error: recv got EOF from 127.12.8.193:52977 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 02:30:37 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:37 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:37.370712 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:37.360666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:48433)
0812 02:30:37.361524 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:37.361548 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:37.361719 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:37.362174 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:37.362179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:37.362679 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:37.362900 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:37.362910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.364138 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.364143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:37.365267 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:37.365282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.366114 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.366120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:37.366297 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:37.366967 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:37.366986 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:37.370317 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":393,"threads_started":1}
W20260812 02:30:37.370914 23646 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:37.360929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:48433 (local address 127.12.8.254:46453)
0812 02:30:37.361283 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:37.361290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:37.361645 (+ 355us) server_negotiation.cc:407] Connection header received
0812 02:30:37.361871 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:37.361875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:37.361964 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:37.362065 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:37.363125 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.363969 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.365435 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.365939 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.366067 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:37.370608 (+ 4541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:48433: BlockingRecv error: recv got EOF from 127.12.8.195:48433 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Aug 12 02:30:37 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:37 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:37.886196 23649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:37.876513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:43989)
0812 02:30:37.877172 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:37.877219 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:37.877369 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:37.877740 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:37.877744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:37.878276 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:37.878471 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:37.878481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.879818 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.879822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:37.880787 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:37.880798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.881586 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.881592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:37.881730 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:37.882431 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:37.882454 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:37.885903 (+ 3449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":343,"threads_started":1}
W20260812 02:30:37.886698 23650 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:37.876590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:43989 (local address 127.12.8.254:46453)
0812 02:30:37.877443 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:37.877448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:37.877464 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:30:37.877511 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:37.877515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:37.877597 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:37.877693 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:37.878635 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.879678 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.880903 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:37.881417 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:37.881514 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:37.886227 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:43989: BlockingRecv error: recv got EOF from 127.12.8.194:43989 (error 108)
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":642,"threads_started":1}
Aug 12 02:30:38 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:38 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:38.296120 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:38.286054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:47657)
0812 02:30:38.286722 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:38.286742 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:38.286906 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:38.287304 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:38.287308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:38.287822 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:38.288039 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:38.288051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.289552 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.289557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:38.290757 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:38.290775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.291533 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.291538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:38.291691 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:38.292370 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:38.292389 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:38.295798 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":354,"threads_started":1}
W20260812 02:30:38.296381 23650 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:38.286216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:47657 (local address 127.12.8.254:46453)
0812 02:30:38.286440 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:38.286446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:38.286866 (+ 420us) server_negotiation.cc:407] Connection header received
0812 02:30:38.287022 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:38.287027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:38.287106 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:38.287209 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:38.288255 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.289369 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.290899 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.291380 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.291485 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:38.296115 (+ 4630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:47657: BlockingRecv error: recv got EOF from 127.12.8.193:47657 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 12 02:30:38 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:38 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:38.382400 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:38.372145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:38193)
0812 02:30:38.372857 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:38.372877 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:38.373070 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:38.373576 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:38.373583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:38.374123 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:38.374330 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:38.374346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.375545 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.375550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:38.376583 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:38.376598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.377597 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.377605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:38.377797 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:38.378613 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:38.378636 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:38.382081 (+ 3445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":392,"threads_started":1}
W20260812 02:30:38.382799 23650 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:38.372323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:38193 (local address 127.12.8.254:46453)
0812 02:30:38.372607 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:38.372616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:38.372993 (+ 377us) server_negotiation.cc:407] Connection header received
0812 02:30:38.373274 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:38.373279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:38.373381 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:38.373496 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:38.374499 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.375358 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.376744 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.377330 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.377475 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:38.382343 (+ 4868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:38193: BlockingRecv error: recv got EOF from 127.12.8.195:38193 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 12 02:30:38 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:38 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:38.897607 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:38.887421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:59127)
0812 02:30:38.888089 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:38.888112 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:38.888287 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:38.888745 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:38.888750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:38.889342 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:38.889589 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:38.889602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.890824 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.890829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:38.892153 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:38.892170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.892926 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.892930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:38.893074 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:38.893789 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:38.893810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:38.897304 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":328,"threads_started":1}
W20260812 02:30:38.897977 23655 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:38.887543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:59127 (local address 127.12.8.254:46453)
0812 02:30:38.888392 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:38.888396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:38.888416 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:30:38.888477 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:38.888480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:38.888552 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:38.888656 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:38.889785 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.890603 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.892316 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:38.892750 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:38.892864 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:38.897506 (+ 4642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:59127: BlockingRecv error: recv got EOF from 127.12.8.194:59127 (error 108)
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":577,"threads_started":1}
Aug 12 02:30:39 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:39 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:39.308109 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:39.297537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:41519)
0812 02:30:39.298179 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:39.298202 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:39.298378 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:39.298803 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:39.298808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:39.299323 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:39.299539 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:39.299564 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.300954 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.300959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:39.302117 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:39.302135 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.302969 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.302977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:39.303192 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:39.304183 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:39.304208 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:39.307749 (+ 3541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":355,"threads_started":1}
W20260812 02:30:39.308230 23655 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:39.297734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:41519 (local address 127.12.8.254:46453)
0812 02:30:39.298011 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:39.298018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:39.298363 (+ 345us) server_negotiation.cc:407] Connection header received
0812 02:30:39.298498 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:39.298501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:39.298581 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:39.298722 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:39.299752 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.300747 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.302291 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.302743 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.302916 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:39.308001 (+ 5085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:41519: BlockingRecv error: recv got EOF from 127.12.8.193:41519 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 12 02:30:39 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:39 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:39.394182 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:39.383632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:56931)
0812 02:30:39.384426 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:39.384448 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:39.384620 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:39.385155 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:39.385159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:39.385673 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:39.385937 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:39.385949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.387246 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.387251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:39.388283 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:39.388296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.389116 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.389121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:39.389327 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:39.390112 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:39.390133 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:39.393846 (+ 3713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":387,"threads_started":1}
W20260812 02:30:39.394621 23655 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:39.383753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:56931 (local address 127.12.8.254:46453)
0812 02:30:39.384025 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:39.384049 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:39.384569 (+ 520us) server_negotiation.cc:407] Connection header received
0812 02:30:39.384773 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:39.384778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:39.384916 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:39.385080 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:39.386161 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.387073 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.388474 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.388941 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.389066 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:39.394241 (+ 5175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:56931: BlockingRecv error: recv got EOF from 127.12.8.195:56931 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 12 02:30:39 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:39 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:39.909066 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:39.899026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:42999)
0812 02:30:39.900020 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:39.900050 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:39.900228 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:39.900737 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:39.900742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:39.901268 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:39.901509 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:39.901525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.902925 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.902938 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:39.904119 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:39.904134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.904897 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.904901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:39.905061 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:39.905756 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:39.905776 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:39.908787 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":381,"threads_started":1}
W20260812 02:30:39.909672 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:39.899162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:42999 (local address 127.12.8.254:46453)
0812 02:30:39.899896 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:39.899903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:39.900198 (+ 295us) server_negotiation.cc:407] Connection header received
0812 02:30:39.900382 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:39.900387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:39.900478 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:39.900621 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:39.901710 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.902757 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.904265 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:39.904753 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:39.904865 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:39.909354 (+ 4489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:42999: BlockingRecv error: recv got EOF from 127.12.8.194:42999 (error 108)
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":488,"threads_started":1}
Aug 12 02:30:40 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:40 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:40.324164 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:40.309368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:40423)
0812 02:30:40.310004 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:40.310027 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:40.310244 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:40.310601 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:40.310607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:40.311210 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:40.311439 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:40.311451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.312597 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.312602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:40.313683 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:40.313700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.314480 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.314486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:40.314658 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:40.320241 (+ 5583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:40.320278 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:40.323859 (+ 3581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":322,"threads_started":1}
W20260812 02:30:40.324564 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:40.309635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:40423 (local address 127.12.8.254:46453)
0812 02:30:40.309943 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:40.309951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:40.310172 (+ 221us) server_negotiation.cc:407] Connection header received
0812 02:30:40.310381 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:40.310386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:40.310456 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:40.310548 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:40.311601 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.312451 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.313862 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.314306 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.314427 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:40.324247 (+ 9820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:40423: BlockingRecv error: recv got EOF from 127.12.8.193:40423 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 12 02:30:40 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:40 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:40.405459 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:40.395504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:58037)
0812 02:30:40.396144 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:40.396165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:40.396337 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:40.396708 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:40.396712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:40.397258 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:40.397472 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:40.397482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.398587 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.398592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:40.399691 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:40.399705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.400618 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.400623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:40.400783 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:40.401525 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:40.401547 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:40.405107 (+ 3560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":345,"threads_started":1}
W20260812 02:30:40.405653 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:40.395680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:58037 (local address 127.12.8.254:46453)
0812 02:30:40.395965 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:40.395972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:40.396281 (+ 309us) server_negotiation.cc:407] Connection header received
0812 02:30:40.396456 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:40.396460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:40.396553 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:40.396647 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:40.397643 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.398377 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.399842 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.400421 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.400526 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:40.405360 (+ 4834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:58037: BlockingRecv error: recv got EOF from 127.12.8.195:58037 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Aug 12 02:30:40 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:40 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:40.921979 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:40.910425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:53071)
0812 02:30:40.911237 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:40.911267 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:40.911465 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:40.911891 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:40.911896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:40.912674 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:40.912918 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:40.912931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.914392 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.914399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:40.915541 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:40.915559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.916419 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.916425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:40.916612 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:40.917603 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:40.917631 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:40.921689 (+ 4058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":384,"threads_started":1}
W20260812 02:30:40.922475 23663 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:40.910667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:53071 (local address 127.12.8.254:46453)
0812 02:30:40.911456 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:40.911463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:40.911487 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:30:40.911615 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:40.911621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:40.911718 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:40.911821 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:40.913152 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.914185 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.915743 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:40.916232 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:40.916364 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:40.922147 (+ 5783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:53071: BlockingRecv error: recv got EOF from 127.12.8.194:53071 (error 108)
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":508,"threads_started":1}
Aug 12 02:30:41 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:41 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:41.334913 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:41.325494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:46625)
0812 02:30:41.326165 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:41.326184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:41.326371 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:41.326712 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:41.326718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:41.327177 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:41.327365 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:41.327375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.328422 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.328426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:41.329431 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:41.329445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.330167 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.330173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:41.330350 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:41.331087 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:41.331108 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:41.334608 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":365,"threads_started":1}
W20260812 02:30:41.335418 23663 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:41.325602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:46625 (local address 127.12.8.254:46453)
0812 02:30:41.325914 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:41.325922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:41.326280 (+ 358us) server_negotiation.cc:407] Connection header received
0812 02:30:41.326500 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:41.326504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:41.326570 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:41.326655 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:41.327512 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.328283 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.329573 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.330006 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.330132 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:41.335110 (+ 4978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:46625: BlockingRecv error: recv got EOF from 127.12.8.193:46625 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 12 02:30:41 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:41 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:41.417479 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:41.406741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:57871)
0812 02:30:41.407397 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:41.407417 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:41.407620 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:41.408008 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:41.408013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:41.408527 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:41.408761 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:41.408771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.410051 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.410058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:41.411470 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:41.411500 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.412440 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.412445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:41.412612 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:41.413509 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:41.413535 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:41.417153 (+ 3618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":351,"threads_started":1}
W20260812 02:30:41.417841 23663 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:41.406949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:57871 (local address 127.12.8.254:46453)
0812 02:30:41.407282 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:41.407290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:41.407531 (+ 241us) server_negotiation.cc:407] Connection header received
0812 02:30:41.407758 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:41.407762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:41.407853 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:41.407952 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:41.408933 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.409822 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.411725 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.412255 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.412387 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:41.417537 (+ 5150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:57871: BlockingRecv error: recv got EOF from 127.12.8.195:57871 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Aug 12 02:30:41 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:41 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:41.933542 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:41.923223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:45585)
0812 02:30:41.923871 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:41.923892 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:41.924014 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:41.924750 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:41.924755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:41.925308 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:41.925578 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:41.925594 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.926728 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.926733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:41.927753 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:41.927766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.928504 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.928509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:41.928664 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:41.929419 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:41.929441 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:41.933196 (+ 3755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":328,"threads_started":1}
W20260812 02:30:41.933729 23667 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:41.923275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:45585 (local address 127.12.8.254:46453)
0812 02:30:41.924398 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:41.924403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:41.924418 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:30:41.924463 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:41.924466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:41.924547 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:41.924637 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:41.925757 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.926557 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.927901 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:41.928319 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:41.928429 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:41.933469 (+ 5040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:45585: BlockingRecv error: recv got EOF from 127.12.8.194:45585 (error 108)
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":901,"threads_started":1}
Aug 12 02:30:42 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:42 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:42.345531 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:42.336275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:50023)
0812 02:30:42.336908 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:42.336926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:42.337055 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:42.337467 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:42.337471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:42.337885 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:42.338059 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:42.338069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.339187 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.339192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:42.340177 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:42.340190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.340995 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.341000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:42.341139 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:42.341842 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:42.341862 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:42.345219 (+ 3357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":275,"threads_started":1}
W20260812 02:30:42.345625 23667 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:42.336426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:50023 (local address 127.12.8.254:46453)
0812 02:30:42.336649 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:42.336654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:42.337008 (+ 354us) server_negotiation.cc:407] Connection header received
0812 02:30:42.337227 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:42.337232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:42.337312 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:42.337419 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:42.338211 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.339016 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.340354 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.340780 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.340888 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:42.345436 (+ 4548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:50023: BlockingRecv error: recv got EOF from 127.12.8.193:50023 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 12 02:30:42 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:42 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:42.428088 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:42.418746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:39911)
0812 02:30:42.419512 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:42.419538 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:42.419802 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:42.420267 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:42.420272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:42.420800 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:42.421009 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:42.421020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.422170 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.422176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:42.423085 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:42.423096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.423897 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.423904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:42.424102 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:42.424808 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:42.424827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:42.427755 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":423,"threads_started":1}
W20260812 02:30:42.428272 23667 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:42.418898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:39911 (local address 127.12.8.254:46453)
0812 02:30:42.419143 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:42.419150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:42.419672 (+ 522us) server_negotiation.cc:407] Connection header received
0812 02:30:42.419968 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:42.419975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:42.420089 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:42.420186 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:42.421159 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.422022 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.423230 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.423725 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.423859 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:42.428006 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:39911: BlockingRecv error: recv got EOF from 127.12.8.195:39911 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 02:30:42 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:42 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:42.944367 23671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:42.934809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:34807)
0812 02:30:42.935438 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:42.935458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:42.935588 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:42.936114 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:42.936118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:42.936592 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:42.936810 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:42.936820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.938030 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.938035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:42.939024 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:42.939038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.939836 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.939840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:42.939963 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:42.940646 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:42.940666 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:42.944040 (+ 3374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":341,"threads_started":1}
W20260812 02:30:42.944532 23673 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:42.935061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:34807 (local address 127.12.8.254:46453)
0812 02:30:42.935797 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:42.935802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:42.935821 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:30:42.935880 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:42.935885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:42.935971 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:42.936094 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:42.936979 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.937870 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.939190 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:42.939673 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:42.939783 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:42.944285 (+ 4502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:34807: BlockingRecv error: recv got EOF from 127.12.8.194:34807 (error 108)
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":496,"threads_started":1}
Aug 12 02:30:43 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:43 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:43.357372 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:43.347016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:58695)
0812 02:30:43.347599 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:43.347617 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:43.347857 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:43.348239 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:43.348244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:43.348741 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:43.348962 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:43.348973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.350146 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.350150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:43.351173 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:43.351188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.351994 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.352000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:43.352194 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:43.352961 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:43.352992 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:43.356967 (+ 3975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":301,"threads_started":1}
W20260812 02:30:43.357573 23673 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:43.347101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:58695 (local address 127.12.8.254:46453)
0812 02:30:43.347354 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:43.347362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:43.347731 (+ 369us) server_negotiation.cc:407] Connection header received
0812 02:30:43.348001 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:43.348005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:43.348086 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:43.348179 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:43.349118 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.349964 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.351341 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.351817 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.351941 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:43.357301 (+ 5360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:58695: BlockingRecv error: recv got EOF from 127.12.8.193:58695 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 02:30:43 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:43 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:43.439107 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:43.429580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:44103)
0812 02:30:43.430224 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:43.430243 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:43.430410 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:43.430814 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:43.430819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:43.431429 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:43.431662 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:43.431674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.432822 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.432828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:43.433895 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:43.433909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.434700 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.434707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:43.434881 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:43.435557 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:43.435577 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:43.438806 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":309,"threads_started":1}
W20260812 02:30:43.439355 23673 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:43.429699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:44103 (local address 127.12.8.254:46453)
0812 02:30:43.430125 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:43.430133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:43.430340 (+ 207us) server_negotiation.cc:407] Connection header received
0812 02:30:43.430556 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:43.430561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:43.430657 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:43.430752 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:43.431820 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.432674 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.434050 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.434502 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.434628 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:43.439066 (+ 4438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:44103: BlockingRecv error: recv got EOF from 127.12.8.195:44103 (error 108)
Metrics: {"server-negotiator.queue_time_us":262}
Aug 12 02:30:43 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:43 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:43.954807 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:43.945777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:36483)
0812 02:30:43.946462 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:43.946490 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:43.946670 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:43.947095 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:43.947100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:43.947649 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:43.947863 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:43.947876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.949031 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.949035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:43.950014 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:43.950027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.950741 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.950747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:43.950906 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:43.951529 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:43.951549 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:43.954453 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":309,"threads_started":1}
W20260812 02:30:43.954978 23679 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:43.945909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:36483 (local address 127.12.8.254:46453)
0812 02:30:43.946754 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:43.946761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:43.946781 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:30:43.946846 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:43.946850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:43.946927 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:43.947033 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:43.948030 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.948883 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.950164 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:43.950591 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:43.950700 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:43.954709 (+ 4009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:36483: BlockingRecv error: recv got EOF from 127.12.8.194:36483 (error 108)
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":581,"threads_started":1}
Aug 12 02:30:44 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:44 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.193@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:44.369781 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:44.359259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.193:34797)
0812 02:30:44.359914 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:44.359936 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:44.360124 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:44.360650 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:44.360657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:44.361344 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:44.361607 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:44.361622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.363140 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.363145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:44.364162 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:44.364176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.365161 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.365166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:44.365326 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:44.365976 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:44.365996 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:44.369448 (+ 3452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":361,"threads_started":1}
W20260812 02:30:44.369921 23679 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:44.359421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:34797 (local address 127.12.8.254:46453)
0812 02:30:44.359612 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:44.359617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:44.360089 (+ 472us) server_negotiation.cc:407] Connection header received
0812 02:30:44.360292 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:44.360297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:44.360398 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:44.360545 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:44.361867 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.362977 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.364358 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.364952 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.365121 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:44.369685 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.193:34797: BlockingRecv error: recv got EOF from 127.12.8.193:34797 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 02:30:44 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:44 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.195@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:44.449892 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:44.440307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.195:55419)
0812 02:30:44.441000 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:44.441023 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:44.441166 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:44.441556 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:44.441559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:44.442002 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:44.442223 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:44.442232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.443359 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.443363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:44.444345 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:44.444357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.445084 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.445090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:44.445305 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:44.446048 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:44.446068 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:44.449601 (+ 3533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":421,"threads_started":1}
W20260812 02:30:44.450484 23679 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:44.440456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:55419 (local address 127.12.8.254:46453)
0812 02:30:44.440660 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:44.440665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:44.441130 (+ 465us) server_negotiation.cc:407] Connection header received
0812 02:30:44.441320 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:44.441324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:44.441405 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:44.441514 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:44.442394 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.443192 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.444495 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.444928 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.445053 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:44.449877 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.195:55419: BlockingRecv error: recv got EOF from 127.12.8.195:55419 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 02:30:44 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:30:44 dist-test-slave-vq2q krb5kdc[22355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.194@KRBTEST.COM for oryx/127.12.8.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:30:44.967367 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:44.956583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46453 (local address 127.12.8.194:47551)
0812 02:30:44.957293 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:44.957316 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:44.957463 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:44.957819 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:44.957822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:44.958405 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:44.958685 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:44.958699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.960216 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.960224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:44.961584 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:44.961608 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.962456 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.962461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:44.962613 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:44.963340 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:44.963362 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:44.966990 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.254:46453: Server oryx/127.12.8.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":344,"threads_started":1}
W20260812 02:30:44.967617 23684 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:44.956752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:47551 (local address 127.12.8.254:46453)
0812 02:30:44.957390 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:44.957395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:44.957417 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:30:44.957578 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:44.957583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:44.957681 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:44.957787 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:44.958886 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.959966 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.961797 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:44.962272 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:44.962385 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:44.967342 (+ 4957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.194:47551: BlockingRecv error: recv got EOF from 127.12.8.194:47551 (error 108)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":324,"threads_started":1}
I20260812 02:30:45.256647 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23085
I20260812 02:30:45.273243 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:45.258165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.12.8.193:38651)
0812 02:30:45.259133 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:45.259145 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:45.259189 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:30:45.259329 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:45.259341 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:45.259732 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:45.260026 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:45.262661 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:45.263884 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:45.265468 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:45.266191 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:45.266331 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:45.269487 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:45.269497 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:45.269517 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:45.269565 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:45.272505 (+ 2940us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 02:30:45.272908 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40122: Request ticket server kudu/127.12.8.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":585,"threads_started":1}
W20260812 02:30:45.273367 23686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40122: Request ticket server kudu/127.12.8.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 02:30:45.273517 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23085: Not authorized: Client connection negotiation failed: client connection to 127.12.8.193:38651: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 02:30:45.273836 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 23085
I20260812 02:30:45.301086 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23276
I20260812 02:30:45.316182 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:45.302533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.12.8.194:43109)
0812 02:30:45.303290 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:45.303299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:45.303337 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:30:45.303414 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:45.303422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:45.303726 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:45.303912 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:45.305452 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:45.306667 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:45.308039 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:45.308695 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:45.308811 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:45.311831 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:45.311844 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:45.311864 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:45.311925 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:45.315458 (+ 3533us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 02:30:45.315859 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48512: Request ticket server kudu/127.12.8.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":383,"threads_started":1}
W20260812 02:30:45.316313 23687 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48512: Request ticket server kudu/127.12.8.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 02:30:45.316525 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23276: Not authorized: Client connection negotiation failed: client connection to 127.12.8.194:43109: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 02:30:45.316712 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 23276
I20260812 02:30:45.344084 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23436
I20260812 02:30:45.363176 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:45.345450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.12.8.195:43445)
0812 02:30:45.346065 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:45.346074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:45.346105 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:30:45.346189 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:45.346197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:45.346476 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:45.346678 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:45.348186 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:45.349485 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:45.350960 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:45.351666 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:45.351782 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:45.359384 (+ 7602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:45.359399 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:45.359420 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:45.359481 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:45.362506 (+ 3025us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 02:30:45.362888 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55728: Request ticket server kudu/127.12.8.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":329,"threads_started":1}
W20260812 02:30:45.363310 23688 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55728: Request ticket server kudu/127.12.8.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 02:30:45.363576 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23436: Not authorized: Client connection negotiation failed: client connection to 127.12.8.195:43445: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 02:30:45.363793 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 23436
I20260812 02:30:45.386641 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23000
I20260812 02:30:45.388451 23077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:45.529259 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 23000
2026-08-12T02:30:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22537 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T02:30:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:30:45Z Disabled control of system clock
I20260812 02:30:45.652930 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:39127
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:39127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 02:30:46.052392 23693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:46.052723 23693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:46.052806 23693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:46.060688 23693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:46.060842 23693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:46.060884 23693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:46.060949 23693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:46.060999 23693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:46.071055 23693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:39127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:39127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:46.072523 23693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:46.074437 23693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:46.088001 23699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:46.087924 23701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:46.087977 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:46.089346 23693 server_base.cc:1034] running on GCE node
I20260812 02:30:46.090255 23693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:46.091922 23693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:46.093200 23693 hybrid_clock.cc:648] HybridClock initialized: now 1786501846093118 us; error 56 us; skew 500 ppm
I20260812 02:30:46.093603 23693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:30:46.096374 23693 webserver.cc:460] Webserver started at http://127.12.8.254:40567/ using document root <none> and password file <none>
I20260812 02:30:46.097155 23693 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:46.097318 23693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:46.097666 23693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:46.100864 23693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc"
format_stamp: "Formatted at 2026-08-12 02:30:46 on dist-test-slave-vq2q"
server_key: "86a20697f4f02462280f1827fed7a3eb"
server_key_iv: "d929025ed8d6ea3d7a4d75f98f797824"
server_key_version: "encryptionkey@0"
I20260812 02:30:46.101742 23693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc"
format_stamp: "Formatted at 2026-08-12 02:30:46 on dist-test-slave-vq2q"
server_key: "86a20697f4f02462280f1827fed7a3eb"
server_key_iv: "d929025ed8d6ea3d7a4d75f98f797824"
server_key_version: "encryptionkey@0"
I20260812 02:30:46.107751 23693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:30:46.111943 23711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:46.113741 23693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:30:46.113942 23693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc"
format_stamp: "Formatted at 2026-08-12 02:30:46 on dist-test-slave-vq2q"
server_key: "86a20697f4f02462280f1827fed7a3eb"
server_key_iv: "d929025ed8d6ea3d7a4d75f98f797824"
server_key_version: "encryptionkey@0"
I20260812 02:30:46.114166 23693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:46.135763 23693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:46.136682 23693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:46.136973 23693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:46.157452 23693 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:39127
I20260812 02:30:46.157501 23777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:39127 every 8 connection(s)
I20260812 02:30:46.158876 23693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:30:46.163897 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 23693
I20260812 02:30:46.163988 23778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:46.164119 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:30:46.164867 12323 external_mini_cluster.cc:1428] Setting key ac882cbddeda0e480225320dd4fd89c1
I20260812 02:30:46.182035 23778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc: Bootstrap starting.
I20260812 02:30:46.184633 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:46.167323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.12.8.254:39127)
0812 02:30:46.168804 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:46.168823 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:46.168899 (+ 76us) server_negotiation.cc:407] Connection header received
0812 02:30:46.172074 (+ 3175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:46.172121 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:46.173020 (+ 899us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:46.173667 (+ 647us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:46.174371 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:46.176421 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:46.178018 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:46.179283 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:46.179523 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:46.180987 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:46.181000 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:46.181027 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:46.181941 (+ 914us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:46.182081 (+ 140us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:46.182115 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:46.182261 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:46.182542 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 02:30:46.183029 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":453,"threads_started":1}
I20260812 02:30:46.186944 23778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:46.188576 23778 log.cc:824] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:46.192334 23778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc: No bootstrap required, opened a new log
I20260812 02:30:46.201118 23778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39127 } }
I20260812 02:30:46.201777 23778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:46.201925 23778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534ec1d9e85244b2ab2df2c6f7d2a9fc, State: Initialized, Role: FOLLOWER
I20260812 02:30:46.202746 23778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39127 } }
I20260812 02:30:46.203016 23778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:30:46.203173 23778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:30:46.203353 23778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:46.206058 23778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39127 } }
I20260812 02:30:46.206717 23778 leader_election.cc:302] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 534ec1d9e85244b2ab2df2c6f7d2a9fc; no voters:
I20260812 02:30:46.207475 23778 leader_election.cc:288] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:30:46.207702 23784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:46.208593 23784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [term 1 LEADER]: Becoming Leader. State: Replica: 534ec1d9e85244b2ab2df2c6f7d2a9fc, State: Running, Role: LEADER
I20260812 02:30:46.209316 23784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39127 } }
I20260812 02:30:46.210253 23778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:46.213407 23786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39127 } } }
I20260812 02:30:46.213373 23788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 534ec1d9e85244b2ab2df2c6f7d2a9fc. Latest consensus state: current_term: 1 leader_uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534ec1d9e85244b2ab2df2c6f7d2a9fc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39127 } } }
I20260812 02:30:46.214099 23788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:46.213886 23786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:46.216003 23800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:30:46.222173 23800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:30:46.234038 23800 catalog_manager.cc:1347] Generated new cluster ID: c8fc1c9166ea4f89acddee4aca04a9bd
I20260812 02:30:46.234202 23800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:30:46.256774 23800 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:30:46.258703 23800 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:30:46.272965 23800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 534ec1d9e85244b2ab2df2c6f7d2a9fc: Generated new TSK 0
I20260812 02:30:46.274230 23800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:30:46.284175 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39127
--builtin_ntp_servers=127.12.8.212:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 02:30:46.724351 23812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:46.724756 23812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:46.724905 23812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:46.733579 23812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:46.733738 23812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:46.733902 23812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:30:46.744935 23812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:46.746763 23812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:46.749027 23812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:46.766120 23817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:46.766199 23820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:46.766147 23818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:46.767743 23812 server_base.cc:1034] running on GCE node
I20260812 02:30:46.768666 23812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:46.770042 23812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:46.771287 23812 hybrid_clock.cc:648] HybridClock initialized: now 1786501846771215 us; error 61 us; skew 500 ppm
I20260812 02:30:46.771715 23812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:30:46.774899 23812 webserver.cc:460] Webserver started at http://127.12.8.193:35243/ using document root <none> and password file <none>
I20260812 02:30:46.775996 23812 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:46.776126 23812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:46.776511 23812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:46.780277 23812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "fe0fe82a5e294d06a48fe7dabc52e927"
format_stamp: "Formatted at 2026-08-12 02:30:46 on dist-test-slave-vq2q"
server_key: "2644b46bd296fe3c45eaa89bd001531b"
server_key_iv: "4cbe1bebdd40b7a1ad160fb47b4d9686"
server_key_version: "encryptionkey@0"
I20260812 02:30:46.781332 23812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "fe0fe82a5e294d06a48fe7dabc52e927"
format_stamp: "Formatted at 2026-08-12 02:30:46 on dist-test-slave-vq2q"
server_key: "2644b46bd296fe3c45eaa89bd001531b"
server_key_iv: "4cbe1bebdd40b7a1ad160fb47b4d9686"
server_key_version: "encryptionkey@0"
I20260812 02:30:46.788564 23812 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 02:30:46.793558 23828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:46.795574 23812 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 02:30:46.795804 23812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "fe0fe82a5e294d06a48fe7dabc52e927"
format_stamp: "Formatted at 2026-08-12 02:30:46 on dist-test-slave-vq2q"
server_key: "2644b46bd296fe3c45eaa89bd001531b"
server_key_iv: "4cbe1bebdd40b7a1ad160fb47b4d9686"
server_key_version: "encryptionkey@0"
I20260812 02:30:46.796087 23812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:46.833577 23812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:46.834712 23812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:46.835041 23812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:46.836264 23812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:46.838130 23812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:46.838238 23812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:46.838361 23812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:46.838431 23812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:46.877527 23812 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:37441
I20260812 02:30:46.877622 23963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:37441 every 8 connection(s)
I20260812 02:30:46.879246 23812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:30:46.888108 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 23812
I20260812 02:30:46.888319 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:30:46.889071 12323 external_mini_cluster.cc:1428] Setting key 0c6e9e41f8bcd4166fc082b1fa2b7931
I20260812 02:30:46.892940 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39127
--builtin_ntp_servers=127.12.8.212:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 02:30:46.894901 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:46.883002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:56339 (local address 127.12.8.254:39127)
0812 02:30:46.883783 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:46.883790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:46.885373 (+ 1583us) server_negotiation.cc:407] Connection header received
0812 02:30:46.887523 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:46.887529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:46.887631 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:46.887734 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:46.889118 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:46.889922 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:46.891863 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:46.892287 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:46.892404 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:46.894008 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:46.894015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:46.894018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:46.894039 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:46.894080 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:46.894092 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:46.894168 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:46.894410 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 02:30:46.894520 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":400,"threads_started":1}
I20260812 02:30:46.896047 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:46.883844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:39127 (local address 127.12.8.193:56339)
0812 02:30:46.885144 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:46.885231 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:46.887164 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:46.888041 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:46.888052 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:46.888153 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:46.888932 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:46.888952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:46.890087 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:46.890091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:46.891680 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:46.891693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:46.892529 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:46.892537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:46.892781 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:46.893613 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:30:46.893617 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:46.893665 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:30:46.893797 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:46.894184 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:46.894196 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:46.894204 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:30:46.894378 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 02:30:46.894828 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":361,"threads_started":1}
I20260812 02:30:46.898145 23964 heartbeater.cc:347] Connected to a master server at 127.12.8.254:39127
I20260812 02:30:46.898510 23964 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:46.899390 23964 heartbeater.cc:511] Master 127.12.8.254:39127 requested a full tablet report, sending...
I20260812 02:30:46.901837 23733 ts_manager.cc:194] Registered new tserver with Master: fe0fe82a5e294d06a48fe7dabc52e927 (127.12.8.193:37441)
I20260812 02:30:46.903421 23733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.193:56339
W20260812 02:30:47.337121 23968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:47.337601 23968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:47.337731 23968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:47.346148 23968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:47.346304 23968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:47.346455 23968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:30:47.357270 23968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:47.359100 23968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:47.361635 23968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:47.379558 23973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:47.379585 23974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:47.380002 23976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:47.381539 23968 server_base.cc:1034] running on GCE node
I20260812 02:30:47.382651 23968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:47.384110 23968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:47.385397 23968 hybrid_clock.cc:648] HybridClock initialized: now 1786501847385294 us; error 71 us; skew 500 ppm
I20260812 02:30:47.385982 23968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:30:47.390038 23968 webserver.cc:460] Webserver started at http://127.12.8.194:39701/ using document root <none> and password file <none>
I20260812 02:30:47.391099 23968 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:47.391234 23968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:47.391626 23968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:47.395449 23968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "d13ea2b912474aab817d023957163048"
format_stamp: "Formatted at 2026-08-12 02:30:47 on dist-test-slave-vq2q"
server_key: "e8b500e474620995a6e0063c0bac3140"
server_key_iv: "7d752841bf739343a4bb4302f24b07ec"
server_key_version: "encryptionkey@0"
I20260812 02:30:47.396450 23968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "d13ea2b912474aab817d023957163048"
format_stamp: "Formatted at 2026-08-12 02:30:47 on dist-test-slave-vq2q"
server_key: "e8b500e474620995a6e0063c0bac3140"
server_key_iv: "7d752841bf739343a4bb4302f24b07ec"
server_key_version: "encryptionkey@0"
I20260812 02:30:47.403414 23968 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 02:30:47.408311 23984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:47.410269 23968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:30:47.410507 23968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "d13ea2b912474aab817d023957163048"
format_stamp: "Formatted at 2026-08-12 02:30:47 on dist-test-slave-vq2q"
server_key: "e8b500e474620995a6e0063c0bac3140"
server_key_iv: "7d752841bf739343a4bb4302f24b07ec"
server_key_version: "encryptionkey@0"
I20260812 02:30:47.410811 23968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:47.426421 23968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:47.427523 23968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:47.427858 23968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:47.429118 23968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:47.431016 23968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:47.431135 23968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:47.431267 23968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:47.431344 23968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:47.469661 23968 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:41871
I20260812 02:30:47.469728 24122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:41871 every 8 connection(s)
I20260812 02:30:47.471365 23968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:30:47.477645 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 23968
I20260812 02:30:47.477834 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:30:47.478350 12323 external_mini_cluster.cc:1428] Setting key c29f2ace5e4823bf8cca2c1621861b6a
I20260812 02:30:47.481685 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39127
--builtin_ntp_servers=127.12.8.212:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 02:30:47.486536 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:47.474621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:58111 (local address 127.12.8.254:39127)
0812 02:30:47.475082 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:47.475087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:47.476704 (+ 1617us) server_negotiation.cc:407] Connection header received
0812 02:30:47.478868 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:47.478874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:47.478973 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:47.479072 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:47.480474 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:47.481269 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:47.483389 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:47.483817 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:47.483934 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:47.485647 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:47.485652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:47.485655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:47.485676 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:47.485718 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:47.485743 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:47.485825 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:47.486067 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 02:30:47.486176 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
I20260812 02:30:47.487682 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:47.475300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:39127 (local address 127.12.8.194:58111)
0812 02:30:47.476504 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:47.476558 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:47.478579 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:47.479344 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:47.479352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:47.479454 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:47.480272 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:47.480312 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:47.481420 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:47.481425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:47.483221 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:47.483235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:47.484070 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:47.484078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:47.484337 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:47.485275 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:30:47.485278 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:47.485340 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:30:47.485432 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:47.485850 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:47.485862 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:47.485870 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:30:47.486050 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 02:30:47.486493 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":364,"threads_started":1}
I20260812 02:30:47.489282 24123 heartbeater.cc:347] Connected to a master server at 127.12.8.254:39127
I20260812 02:30:47.489619 24123 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:47.490454 24123 heartbeater.cc:511] Master 127.12.8.254:39127 requested a full tablet report, sending...
I20260812 02:30:47.491882 23733 ts_manager.cc:194] Registered new tserver with Master: d13ea2b912474aab817d023957163048 (127.12.8.194:41871)
I20260812 02:30:47.492669 23733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.194:58111
W20260812 02:30:47.906989 24127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:47.907332 24127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260812 02:30:47.907295 23964 heartbeater.cc:503] Master 127.12.8.254:39127 was elected leader, sending a full tablet report...
W20260812 02:30:47.907513 24127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:47.915539 24127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:47.915685 24127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:47.915879 24127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:47.925622 24127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:47.927142 24127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:47.929304 24127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:47.945310 24134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:47.945351 24136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:47.945765 24139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:47.946782 24127 server_base.cc:1034] running on GCE node
I20260812 02:30:47.947460 24127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:47.948601 24127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:47.949815 24127 hybrid_clock.cc:648] HybridClock initialized: now 1786501847949776 us; error 45 us; skew 500 ppm
I20260812 02:30:47.950215 24127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:30:47.952986 24127 webserver.cc:460] Webserver started at http://127.12.8.195:44463/ using document root <none> and password file <none>
I20260812 02:30:47.953824 24127 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:47.953943 24127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:47.954300 24127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:47.957445 24127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "cba8c95528ed41cea74d63d97e78da66"
format_stamp: "Formatted at 2026-08-12 02:30:47 on dist-test-slave-vq2q"
server_key: "d812c8ba914a493bb8530090eab455d4"
server_key_iv: "c6f1f4d4ff486dc1825fb011604eedfd"
server_key_version: "encryptionkey@0"
I20260812 02:30:47.958298 24127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "cba8c95528ed41cea74d63d97e78da66"
format_stamp: "Formatted at 2026-08-12 02:30:47 on dist-test-slave-vq2q"
server_key: "d812c8ba914a493bb8530090eab455d4"
server_key_iv: "c6f1f4d4ff486dc1825fb011604eedfd"
server_key_version: "encryptionkey@0"
I20260812 02:30:47.964195 24127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:30:47.968201 24147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:47.969870 24127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:30:47.970074 24127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "cba8c95528ed41cea74d63d97e78da66"
format_stamp: "Formatted at 2026-08-12 02:30:47 on dist-test-slave-vq2q"
server_key: "d812c8ba914a493bb8530090eab455d4"
server_key_iv: "c6f1f4d4ff486dc1825fb011604eedfd"
server_key_version: "encryptionkey@0"
I20260812 02:30:47.970292 24127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:47.986761 24127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:47.987671 24127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:47.987972 24127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:47.989065 24127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:47.990767 24127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:47.990871 24127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:47.990999 24127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:47.991071 24127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:48.027848 24127 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:38791
I20260812 02:30:48.027949 24285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:38791 every 8 connection(s)
I20260812 02:30:48.029373 24127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:48.034391 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 24127
I20260812 02:30:48.034602 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:30:48.035243 12323 external_mini_cluster.cc:1428] Setting key f238e290bb60631192792abac09e7ffe
I20260812 02:30:48.045713 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:48.032712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:41061 (local address 127.12.8.254:39127)
0812 02:30:48.033316 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:48.033323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:48.035130 (+ 1807us) server_negotiation.cc:407] Connection header received
0812 02:30:48.037453 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:48.037461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:48.037560 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:48.037676 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:48.039257 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.040131 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.042340 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.042764 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.042890 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:48.044698 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:48.044703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:48.044707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:48.044731 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:48.044781 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:48.044794 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:48.044886 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:48.045290 (+ 404us) server_negotiation.cc:299] Negotiation successful
0812 02:30:48.045404 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":241,"threads_started":1}
I20260812 02:30:48.047123 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:48.033499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:39127 (local address 127.12.8.195:41061)
0812 02:30:48.034917 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:48.034981 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:48.037087 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:48.038051 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:48.038079 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:48.038220 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:48.039022 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:48.039050 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.040331 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.040336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:48.042135 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:48.042152 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.043051 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.043058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:48.043374 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:48.044259 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:30:48.044262 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:48.044319 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:30:48.044439 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:48.044934 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:48.044945 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:48.044953 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:30:48.045255 (+ 302us) client_negotiation.cc:240] Negotiation successful
0812 02:30:48.045707 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":483,"threads_started":1}
I20260812 02:30:48.048900 24287 heartbeater.cc:347] Connected to a master server at 127.12.8.254:39127
I20260812 02:30:48.049325 24287 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:48.050449 24287 heartbeater.cc:511] Master 127.12.8.254:39127 requested a full tablet report, sending...
I20260812 02:30:48.052409 23731 ts_manager.cc:194] Registered new tserver with Master: cba8c95528ed41cea74d63d97e78da66 (127.12.8.195:38791)
I20260812 02:30:48.053408 23731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.195:41061
I20260812 02:30:48.065371 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:30:48.072650 24289 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:48.070906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.12.8.254:39127)
0812 02:30:48.071155 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:48.071161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:48.071472 (+ 311us) server_negotiation.cc:407] Connection header received
0812 02:30:48.071614 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:48.071619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:48.071709 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:48.071845 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:48.072371 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52466: BlockingRecv error: recv got EOF from 127.0.0.1:52466 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:30:48.073588 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23812
I20260812 02:30:48.083259 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:48.075045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.12.8.193:37441)
0812 02:30:48.075786 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:48.075795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:48.075827 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:30:48.075912 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:48.075920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:48.076215 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:48.076389 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:48.076889 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.078021 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.079427 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.080127 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.080259 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:48.081549 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:48.081559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:48.081576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:48.082296 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:48.082357 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:48.082380 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:48.082506 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:48.082657 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 02:30:48.082827 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":316,"threads_started":1}
I20260812 02:30:48.084786 23959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:48.172489 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 23812
I20260812 02:30:48.195508 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23968
I20260812 02:30:48.205302 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:48.196646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.12.8.194:41871)
0812 02:30:48.197387 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:48.197397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:48.197431 (+ 34us) server_negotiation.cc:407] Connection header received
0812 02:30:48.197523 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:48.197534 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:48.197828 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:48.198011 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:48.198497 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.199634 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.201284 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.202016 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.202158 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:48.203417 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:48.203426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:48.203444 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:48.204239 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:48.204302 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:48.204325 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:48.204459 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:48.204633 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 02:30:48.204836 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":336,"threads_started":1}
I20260812 02:30:48.206910 24117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:48.297852 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 23968
I20260812 02:30:48.324487 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 24127
I20260812 02:30:48.334719 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:48.325826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.12.8.195:38791)
0812 02:30:48.326707 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:48.326716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:48.326756 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:30:48.326841 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:48.326848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:48.327142 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:48.327318 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:48.327840 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.329067 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.330536 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:48.331304 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:48.331443 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:48.332706 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:48.332715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:48.332733 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:48.333603 (+ 870us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:48.333672 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:48.333697 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:48.333803 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:48.334072 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 02:30:48.334273 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":451,"threads_started":1}
I20260812 02:30:48.336506 24278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:48.430254 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 24127
I20260812 02:30:48.449595 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 23693
I20260812 02:30:48.451160 23772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:48.572352 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 23693
2026-08-12T02:30:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3034 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T02:30:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:30:48Z Disabled control of system clock
I20260812 02:30:48.679936 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:39173
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:39173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 02:30:49.080333 24315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:49.080669 24315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:49.080758 24315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:49.088238 24315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:49.088382 24315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:49.088424 24315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:49.088486 24315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:49.088536 24315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:49.098147 24315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:39173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:49.099545 24315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:49.101501 24315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:49.114470 24320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:49.114531 24323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:49.114467 24321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:49.116154 24315 server_base.cc:1034] running on GCE node
I20260812 02:30:49.117004 24315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:49.118649 24315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:49.119880 24315 hybrid_clock.cc:648] HybridClock initialized: now 1786501849119797 us; error 84 us; skew 500 ppm
I20260812 02:30:49.120282 24315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:30:49.122915 24315 webserver.cc:460] Webserver started at http://127.12.8.254:38555/ using document root <none> and password file <none>
I20260812 02:30:49.123678 24315 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:49.123786 24315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:49.124140 24315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:49.127444 24315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "224661ed236844afbf5e5b6d1535c415"
format_stamp: "Formatted at 2026-08-12 02:30:49 on dist-test-slave-vq2q"
server_key: "1b24eb9c617cdba2058149993d8a5d61"
server_key_iv: "75047ad96ea118bfa0b39be3dd256892"
server_key_version: "encryptionkey@0"
I20260812 02:30:49.128268 24315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "224661ed236844afbf5e5b6d1535c415"
format_stamp: "Formatted at 2026-08-12 02:30:49 on dist-test-slave-vq2q"
server_key: "1b24eb9c617cdba2058149993d8a5d61"
server_key_iv: "75047ad96ea118bfa0b39be3dd256892"
server_key_version: "encryptionkey@0"
I20260812 02:30:49.134416 24315 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 02:30:49.138732 24329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:49.140763 24315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 02:30:49.140980 24315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "224661ed236844afbf5e5b6d1535c415"
format_stamp: "Formatted at 2026-08-12 02:30:49 on dist-test-slave-vq2q"
server_key: "1b24eb9c617cdba2058149993d8a5d61"
server_key_iv: "75047ad96ea118bfa0b39be3dd256892"
server_key_version: "encryptionkey@0"
I20260812 02:30:49.141248 24315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:49.147732 24315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:49.148561 24315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:49.148849 24315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:49.169708 24315 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:39173
I20260812 02:30:49.169828 24394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:39173 every 8 connection(s)
I20260812 02:30:49.171283 24315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:30:49.176477 24395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:49.178102 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 24315
I20260812 02:30:49.178368 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:30:49.179343 12323 external_mini_cluster.cc:1428] Setting key 310ec1b64b56f1882fab63b317a0774b
I20260812 02:30:49.190378 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:49.182203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.12.8.254:39173)
0812 02:30:49.183739 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:49.183753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:49.183809 (+ 56us) server_negotiation.cc:407] Connection header received
0812 02:30:49.185909 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:49.185936 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:49.186453 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:49.186850 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:49.188338 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:49.188346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:49.188359 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:49.188377 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:49.188460 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:49.188482 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:49.188704 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:49.188831 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 02:30:49.189165 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":617,"threads_started":1}
I20260812 02:30:49.192659 24395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415: Bootstrap starting.
I20260812 02:30:49.196802 24395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:49.198187 24395 log.cc:824] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:49.201978 24395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415: No bootstrap required, opened a new log
I20260812 02:30:49.210561 24395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224661ed236844afbf5e5b6d1535c415" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39173 } }
I20260812 02:30:49.211071 24395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:49.211196 24395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224661ed236844afbf5e5b6d1535c415, State: Initialized, Role: FOLLOWER
I20260812 02:30:49.211951 24395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224661ed236844afbf5e5b6d1535c415" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39173 } }
I20260812 02:30:49.212219 24395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:30:49.212386 24395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:30:49.212558 24395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:49.215405 24395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224661ed236844afbf5e5b6d1535c415" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39173 } }
I20260812 02:30:49.216133 24395 leader_election.cc:302] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 224661ed236844afbf5e5b6d1535c415; no voters:
I20260812 02:30:49.216889 24395 leader_election.cc:288] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:30:49.217092 24400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:49.218104 24400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [term 1 LEADER]: Becoming Leader. State: Replica: 224661ed236844afbf5e5b6d1535c415, State: Running, Role: LEADER
I20260812 02:30:49.218844 24400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224661ed236844afbf5e5b6d1535c415" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39173 } }
I20260812 02:30:49.219686 24395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:49.222267 24402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 224661ed236844afbf5e5b6d1535c415. Latest consensus state: current_term: 1 leader_uuid: "224661ed236844afbf5e5b6d1535c415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224661ed236844afbf5e5b6d1535c415" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39173 } } }
I20260812 02:30:49.222350 24401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "224661ed236844afbf5e5b6d1535c415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224661ed236844afbf5e5b6d1535c415" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 39173 } } }
I20260812 02:30:49.222994 24402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:49.222992 24401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415 [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:49.223935 24408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:30:49.229586 24408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:30:49.239163 24408 catalog_manager.cc:1347] Generated new cluster ID: f8a88a0b55f5421c924d31dc0fb9f949
I20260812 02:30:49.239312 24408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:30:49.253129 24408 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:30:49.253687 24408 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:30:49.261284 24408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 224661ed236844afbf5e5b6d1535c415: Generated new TSK 0
I20260812 02:30:49.262193 24408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:30:49.267552 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39173
--builtin_ntp_servers=127.12.8.212:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 02:30:49.638973 24424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:49.639278 24424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:49.639392 24424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:49.647202 24424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:49.647333 24424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:49.647485 24424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:30:49.657200 24424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:49.658587 24424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:49.660414 24424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:49.675163 24429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:49.675201 24430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:49.676120 24432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:49.676209 24424 server_base.cc:1034] running on GCE node
I20260812 02:30:49.676936 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:49.678053 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:49.679289 24424 hybrid_clock.cc:648] HybridClock initialized: now 1786501849679213 us; error 62 us; skew 500 ppm
I20260812 02:30:49.679656 24424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:30:49.682310 24424 webserver.cc:460] Webserver started at http://127.12.8.193:34827/ using document root <none> and password file <none>
I20260812 02:30:49.683048 24424 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:49.683142 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:49.683480 24424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:49.686594 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "124dd68247b14516801b13ba377c0a60"
format_stamp: "Formatted at 2026-08-12 02:30:49 on dist-test-slave-vq2q"
server_key: "b5408c6a690dffe1b88a51257b801946"
server_key_iv: "f3ee4b5c1d63d31fba5e11dcc4013752"
server_key_version: "encryptionkey@0"
I20260812 02:30:49.687402 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "124dd68247b14516801b13ba377c0a60"
format_stamp: "Formatted at 2026-08-12 02:30:49 on dist-test-slave-vq2q"
server_key: "b5408c6a690dffe1b88a51257b801946"
server_key_iv: "f3ee4b5c1d63d31fba5e11dcc4013752"
server_key_version: "encryptionkey@0"
I20260812 02:30:49.693634 24424 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 02:30:49.697980 24439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:49.699728 24424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:30:49.699952 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "124dd68247b14516801b13ba377c0a60"
format_stamp: "Formatted at 2026-08-12 02:30:49 on dist-test-slave-vq2q"
server_key: "b5408c6a690dffe1b88a51257b801946"
server_key_iv: "f3ee4b5c1d63d31fba5e11dcc4013752"
server_key_version: "encryptionkey@0"
I20260812 02:30:49.700181 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:49.706557 24424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:49.707448 24424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:49.707715 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:49.708767 24424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:49.710505 24424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:49.710609 24424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:49.710721 24424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:49.710794 24424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:49.748659 24424 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:40091
I20260812 02:30:49.748760 24577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:40091 every 8 connection(s)
I20260812 02:30:49.750166 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:30:49.752723 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 24424
I20260812 02:30:49.753010 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:30:49.753885 12323 external_mini_cluster.cc:1428] Setting key 9f6aa6404327d5cb92a07b0f51aa336c
I20260812 02:30:49.757973 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39173
--builtin_ntp_servers=127.12.8.212:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 02:30:49.762174 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:49.753839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:35869 (local address 127.12.8.254:39173)
0812 02:30:49.754598 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:49.754606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:49.756051 (+ 1445us) server_negotiation.cc:407] Connection header received
0812 02:30:49.758905 (+ 2854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:49.758910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:49.758965 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:49.759061 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:49.761284 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:49.761289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:49.761292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:49.761331 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:49.761406 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:49.761426 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:49.761604 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:49.761891 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 02:30:49.761949 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":418,"threads_started":1}
I20260812 02:30:49.763938 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:49.754379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:39173 (local address 127.12.8.193:35869)
0812 02:30:49.755841 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:49.755894 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:49.758470 (+ 2576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:49.759453 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:49.759471 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:49.759670 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:49.760709 (+ 1039us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:30:49.760713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:49.760790 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:30:49.760954 (+ 164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:49.761571 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:49.761585 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:49.761597 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 02:30:49.761986 (+ 389us) client_negotiation.cc:240] Negotiation successful
0812 02:30:49.762378 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":413,"threads_started":1}
I20260812 02:30:49.765709 24578 heartbeater.cc:347] Connected to a master server at 127.12.8.254:39173
I20260812 02:30:49.766172 24578 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:49.767102 24578 heartbeater.cc:511] Master 127.12.8.254:39173 requested a full tablet report, sending...
I20260812 02:30:49.769860 24352 ts_manager.cc:194] Registered new tserver with Master: 124dd68247b14516801b13ba377c0a60 (127.12.8.193:40091)
W20260812 02:30:50.120420 24583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:50.120756 24583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:50.120883 24583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:50.128494 24583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:50.128626 24583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:50.128767 24583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:30:50.138779 24583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:50.140255 24583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:50.142218 24583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:50.156910 24590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:50.156836 24594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:50.156971 24591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:50.158272 24583 server_base.cc:1034] running on GCE node
I20260812 02:30:50.158934 24583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:50.159979 24583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:50.161250 24583 hybrid_clock.cc:648] HybridClock initialized: now 1786501850161187 us; error 55 us; skew 500 ppm
I20260812 02:30:50.161619 24583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:30:50.164131 24583 webserver.cc:460] Webserver started at http://127.12.8.194:33369/ using document root <none> and password file <none>
I20260812 02:30:50.164880 24583 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:50.165004 24583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:50.165409 24583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:50.168550 24583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "d570f5597eee430bb750081e8f58a19d"
format_stamp: "Formatted at 2026-08-12 02:30:50 on dist-test-slave-vq2q"
server_key: "1aaba62a2f41c6290e29014d35ac9708"
server_key_iv: "58b64b41ca98581db0d07217897052da"
server_key_version: "encryptionkey@0"
I20260812 02:30:50.169485 24583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "d570f5597eee430bb750081e8f58a19d"
format_stamp: "Formatted at 2026-08-12 02:30:50 on dist-test-slave-vq2q"
server_key: "1aaba62a2f41c6290e29014d35ac9708"
server_key_iv: "58b64b41ca98581db0d07217897052da"
server_key_version: "encryptionkey@0"
I20260812 02:30:50.175540 24583 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:30:50.179757 24600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:50.181677 24583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 02:30:50.181874 24583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "d570f5597eee430bb750081e8f58a19d"
format_stamp: "Formatted at 2026-08-12 02:30:50 on dist-test-slave-vq2q"
server_key: "1aaba62a2f41c6290e29014d35ac9708"
server_key_iv: "58b64b41ca98581db0d07217897052da"
server_key_version: "encryptionkey@0"
I20260812 02:30:50.182111 24583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:50.188136 24583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:50.188978 24583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:50.189293 24583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:50.190363 24583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:50.192011 24583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:50.192116 24583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:50.192241 24583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:50.192315 24583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:50.230052 24583 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:33601
I20260812 02:30:50.230168 24738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:33601 every 8 connection(s)
I20260812 02:30:50.231495 24583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:30:50.241806 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 24583
I20260812 02:30:50.242208 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:30:50.242874 12323 external_mini_cluster.cc:1428] Setting key 30818c00056bec0324032b671f86bd22
I20260812 02:30:50.243332 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:50.235098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:50573 (local address 127.12.8.254:39173)
0812 02:30:50.235398 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:50.235405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:50.237309 (+ 1904us) server_negotiation.cc:407] Connection header received
0812 02:30:50.239926 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:50.239931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:50.240001 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:50.240123 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:50.242275 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:50.242315 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:50.242320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:50.242356 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:50.242421 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:50.242468 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:50.242675 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:50.242915 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 02:30:50.243008 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260812 02:30:50.244517 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:50.235631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:39173 (local address 127.12.8.194:50573)
0812 02:30:50.237023 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:50.237077 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:50.239456 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:50.240539 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:50.240552 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:50.240707 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:50.241637 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:30:50.241641 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:50.241693 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:30:50.241834 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:50.242650 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:50.242661 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:50.242670 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:30:50.242976 (+ 306us) client_negotiation.cc:240] Negotiation successful
0812 02:30:50.243279 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":467,"threads_started":1}
I20260812 02:30:50.246520 24740 heartbeater.cc:347] Connected to a master server at 127.12.8.254:39173
I20260812 02:30:50.246860 24740 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:50.246886 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39173
--builtin_ntp_servers=127.12.8.212:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 02:30:50.247519 24740 heartbeater.cc:511] Master 127.12.8.254:39173 requested a full tablet report, sending...
I20260812 02:30:50.249503 24352 ts_manager.cc:194] Registered new tserver with Master: d570f5597eee430bb750081e8f58a19d (127.12.8.194:33601)
W20260812 02:30:50.602326 24744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:50.602638 24744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:50.602742 24744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:50.610425 24744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:50.610562 24744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:50.610697 24744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:50.620332 24744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:50.621863 24744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:50.623685 24744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:50.638497 24753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:50.638741 24751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:50.639483 24744 server_base.cc:1034] running on GCE node
W20260812 02:30:50.639993 24750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:50.640987 24744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:50.642421 24744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:50.643769 24744 hybrid_clock.cc:648] HybridClock initialized: now 1786501850643684 us; error 72 us; skew 500 ppm
I20260812 02:30:50.644253 24744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:30:50.659737 24744 webserver.cc:460] Webserver started at http://127.12.8.195:46451/ using document root <none> and password file <none>
I20260812 02:30:50.660730 24744 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:50.660862 24744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:50.661300 24744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:50.665012 24744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "571ded0acf7a4bfdaa78af1bac2da267"
format_stamp: "Formatted at 2026-08-12 02:30:50 on dist-test-slave-vq2q"
server_key: "57e33a2bab6daabd32fdbb81d4a2d103"
server_key_iv: "de393b0c2f5ab6ab4d5d6bac8a1401eb"
server_key_version: "encryptionkey@0"
I20260812 02:30:50.666090 24744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "571ded0acf7a4bfdaa78af1bac2da267"
format_stamp: "Formatted at 2026-08-12 02:30:50 on dist-test-slave-vq2q"
server_key: "57e33a2bab6daabd32fdbb81d4a2d103"
server_key_iv: "de393b0c2f5ab6ab4d5d6bac8a1401eb"
server_key_version: "encryptionkey@0"
I20260812 02:30:50.672878 24744 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 02:30:50.677584 24763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:50.679595 24744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:30:50.679844 24744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "571ded0acf7a4bfdaa78af1bac2da267"
format_stamp: "Formatted at 2026-08-12 02:30:50 on dist-test-slave-vq2q"
server_key: "57e33a2bab6daabd32fdbb81d4a2d103"
server_key_iv: "de393b0c2f5ab6ab4d5d6bac8a1401eb"
server_key_version: "encryptionkey@0"
I20260812 02:30:50.680121 24744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:50.688124 24744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:50.689314 24744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:50.689633 24744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:50.690855 24744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:50.692786 24744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:50.692912 24744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:50.693044 24744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:50.693118 24744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:50.734298 24744 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:43629
I20260812 02:30:50.734392 24904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:43629 every 8 connection(s)
I20260812 02:30:50.736169 24744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:50.744779 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 24744
I20260812 02:30:50.745052 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:30:50.745689 12323 external_mini_cluster.cc:1428] Setting key 7dc910018147809718d791abfe88fb29
I20260812 02:30:50.746650 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:50.739691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:39835 (local address 127.12.8.254:39173)
0812 02:30:50.740017 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:50.740026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:50.741918 (+ 1892us) server_negotiation.cc:407] Connection header received
0812 02:30:50.743846 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:50.743852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:50.743932 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:50.744087 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:50.745610 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:50.745619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:50.745625 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:50.745653 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:50.745713 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:50.745746 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:50.745978 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:50.746166 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 02:30:50.746251 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 02:30:50.747641 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:50.740422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:39173 (local address 127.12.8.195:39835)
0812 02:30:50.741697 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:50.741745 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:50.743524 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:50.744305 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:50.744313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:50.744409 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:50.745222 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:30:50.745226 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:50.745283 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:30:50.745388 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:50.745935 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:50.745946 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:50.745955 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:30:50.746264 (+ 309us) client_negotiation.cc:240] Negotiation successful
0812 02:30:50.746522 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":378,"threads_started":1}
I20260812 02:30:50.749701 24905 heartbeater.cc:347] Connected to a master server at 127.12.8.254:39173
I20260812 02:30:50.750142 24905 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:50.750996 24905 heartbeater.cc:511] Master 127.12.8.254:39173 requested a full tablet report, sending...
I20260812 02:30:50.752939 24352 ts_manager.cc:194] Registered new tserver with Master: 571ded0acf7a4bfdaa78af1bac2da267 (127.12.8.195:43629)
I20260812 02:30:50.763935 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:30:50.771315 24580 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:30:50.769576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.12.8.254:39173)
0812 02:30:50.769807 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:50.769812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:50.770360 (+ 548us) server_negotiation.cc:407] Connection header received
0812 02:30:50.770516 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:50.770520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:50.770587 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:50.770698 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:50.771073 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46600: BlockingRecv error: recv got EOF from 127.0.0.1:46600 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 02:30:50.772387 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 24424
I20260812 02:30:50.774051 24578 heartbeater.cc:503] Master 127.12.8.254:39173 was elected leader, sending a full tablet report...
I20260812 02:30:50.777678 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:50.774169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.12.8.193:40091)
0812 02:30:50.774893 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:50.774904 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:50.774941 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:30:50.775028 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:50.775039 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:50.775350 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:50.775514 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:50.776734 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:50.776741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:50.776756 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:50.776770 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:50.776820 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:50.776836 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:50.776999 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:50.777102 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 02:30:50.777305 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":334,"threads_started":1}
I20260812 02:30:50.779165 24573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:50.874576 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 24424
I20260812 02:30:50.895784 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 24583
I20260812 02:30:50.900641 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:50.897025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.12.8.194:33601)
0812 02:30:50.897783 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:50.897791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:50.897828 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:30:50.897918 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:50.897928 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:50.898236 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:50.898407 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:50.899818 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:50.899827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:50.899843 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:50.899867 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:50.899922 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:50.899941 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:50.900129 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:50.900211 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:30:50.900323 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":375,"threads_started":1}
I20260812 02:30:50.902345 24733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:50.992995 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 24583
I20260812 02:30:51.016319 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 24744
I20260812 02:30:51.021042 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:51.017533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.12.8.195:43629)
0812 02:30:51.018202 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:51.018211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:51.018247 (+ 36us) server_negotiation.cc:407] Connection header received
0812 02:30:51.018327 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:51.018334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:51.018645 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:51.018815 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:51.020153 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:51.020160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:51.020175 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:30:51.020189 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:51.020240 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:30:51.020256 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:51.020439 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:51.020513 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 02:30:51.020663 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":299,"threads_started":1}
I20260812 02:30:51.022686 24899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:51.113904 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 24744
I20260812 02:30:51.139284 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 24315
I20260812 02:30:51.140858 24388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:51.260082 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 24315
2026-08-12T02:30:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2690 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:30:51 dist-test-slave-vq2q krb5kdc[24925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:30:51 dist-test-slave-vq2q krb5kdc[24925](info): set up 2 sockets
Aug 12 02:30:51 dist-test-slave-vq2q krb5kdc[24925](info): commencing operation
krb5kdc: starting...
W20260812 02:30:52.448427 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:30:52 dist-test-slave-vq2q krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:30:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:30:52.806164 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33497
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:33497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:53.190187 24941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:53.190508 24941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:53.190631 24941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:53.197983 24941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:30:53.198107 24941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:53.198155 24941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:53.198218 24941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:30:53.198287 24941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:30:53.208503 24941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:33497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:53.210012 24941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:53.211424 24941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:53.224987 24950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:53.225013 24948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:53.225303 24946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:53.225952 24941 server_base.cc:1034] running on GCE node
I20260812 02:30:53.226864 24941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:53.228519 24941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:53.229770 24941 hybrid_clock.cc:648] HybridClock initialized: now 1786501853229693 us; error 60 us; skew 500 ppm
I20260812 02:30:53.230163 24941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:53 dist-test-slave-vq2q krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501853, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:53.234023 24941 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:30:53.235836 24941 webserver.cc:460] Webserver started at http://127.12.8.254:35379/ using document root <none> and password file <none>
I20260812 02:30:53.236613 24941 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:53.236748 24941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:53.237092 24941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:53.240417 24941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "04c5ab9b062249ac94890599ca0663e5"
format_stamp: "Formatted at 2026-08-12 02:30:53 on dist-test-slave-vq2q"
server_key: "1c1846c73917f760cffd11cb9d169292"
server_key_iv: "8dada095b3315546d31730ef77051c19"
server_key_version: "encryptionkey@0"
I20260812 02:30:53.241282 24941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "04c5ab9b062249ac94890599ca0663e5"
format_stamp: "Formatted at 2026-08-12 02:30:53 on dist-test-slave-vq2q"
server_key: "1c1846c73917f760cffd11cb9d169292"
server_key_iv: "8dada095b3315546d31730ef77051c19"
server_key_version: "encryptionkey@0"
I20260812 02:30:53.247611 24941 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:30:53.251825 24957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:53.253552 24941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:30:53.253750 24941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "04c5ab9b062249ac94890599ca0663e5"
format_stamp: "Formatted at 2026-08-12 02:30:53 on dist-test-slave-vq2q"
server_key: "1c1846c73917f760cffd11cb9d169292"
server_key_iv: "8dada095b3315546d31730ef77051c19"
server_key_version: "encryptionkey@0"
I20260812 02:30:53.253989 24941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:53.274657 24941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:53.279330 24941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:53.279621 24941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:53.301868 24941 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:33497
I20260812 02:30:53.301978 25018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:33497 every 8 connection(s)
I20260812 02:30:53.303347 24941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:30:53.305253 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 24941
I20260812 02:30:53.305529 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:30:53.306463 12323 external_mini_cluster.cc:1428] Setting key 36326ced133ddd4ae5d73be1b73cb8b8
I20260812 02:30:53.309031 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:53.325742 25019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5: Bootstrap starting.
Aug 12 02:30:53 dist-test-slave-vq2q krb5kdc[24925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:53.330422 25019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:53.332134 25019 log.cc:824] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:53.336720 25019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5: No bootstrap required, opened a new log
I20260812 02:30:53.338424 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:53.309203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.12.8.254:33497)
0812 02:30:53.310301 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:53.310320 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:53.313913 (+ 3593us) server_negotiation.cc:407] Connection header received
0812 02:30:53.316039 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:53.316080 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:53.316817 (+ 737us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:53.317452 (+ 635us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:53.318970 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:53.321048 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:53.322527 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:53.323731 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:53.323963 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:53.328561 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:53.328574 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:53.328596 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:53.328670 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:53.333125 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:53.334089 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:53.334098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:53.334107 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:53.334356 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:53.334723 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:53.334727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:53.334729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:53.335526 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:53.335883 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:53.336263 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 02:30:53.336787 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":459,"threads_started":1}
I20260812 02:30:53.345803 25019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c5ab9b062249ac94890599ca0663e5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33497 } }
I20260812 02:30:53.346306 25019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:53.346423 25019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c5ab9b062249ac94890599ca0663e5, State: Initialized, Role: FOLLOWER
I20260812 02:30:53.347124 25019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c5ab9b062249ac94890599ca0663e5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33497 } }
I20260812 02:30:53.347366 25019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:30:53.347512 25019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:30:53.347680 25019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:53.350399 25019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c5ab9b062249ac94890599ca0663e5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33497 } }
I20260812 02:30:53.351009 25019 leader_election.cc:302] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c5ab9b062249ac94890599ca0663e5; no voters:
I20260812 02:30:53.351697 25019 leader_election.cc:288] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:30:53.351877 25025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:53.352839 25025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [term 1 LEADER]: Becoming Leader. State: Replica: 04c5ab9b062249ac94890599ca0663e5, State: Running, Role: LEADER
I20260812 02:30:53.353538 25025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c5ab9b062249ac94890599ca0663e5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33497 } }
I20260812 02:30:53.354269 25019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:30:53.357901 25027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c5ab9b062249ac94890599ca0663e5. Latest consensus state: current_term: 1 leader_uuid: "04c5ab9b062249ac94890599ca0663e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c5ab9b062249ac94890599ca0663e5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33497 } } }
I20260812 02:30:53.357575 25026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04c5ab9b062249ac94890599ca0663e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c5ab9b062249ac94890599ca0663e5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33497 } } }
I20260812 02:30:53.359140 25027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:53.359277 25026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:30:53.366025 25034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:30:53.372434 25034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:30:53.384342 25034 catalog_manager.cc:1347] Generated new cluster ID: b1ab12871f9f4f97b3d9b1e72a6d5106
I20260812 02:30:53.384500 25034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:30:53.394255 25034 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:30:53.396437 25034 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:30:53.406556 25034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5: Generated new TSK 0
I20260812 02:30:53.407897 25034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:30:53.505546 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33497
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:53.947389 25051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:53.947778 25051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:53.947890 25051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:53.956528 25051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:53.956689 25051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:53.956825 25051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:30:53.968945 25051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:53.970716 25051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:53.972406 25051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:53.989583 25058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:53.989866 25057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:53.989651 25061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:53.990856 25051 server_base.cc:1034] running on GCE node
I20260812 02:30:53.991951 25051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:53.993237 25051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:53.994522 25051 hybrid_clock.cc:648] HybridClock initialized: now 1786501853994463 us; error 61 us; skew 500 ppm
I20260812 02:30:53.994925 25051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:53 dist-test-slave-vq2q krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501853, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:53.998888 25051 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:30:54.000690 25051 webserver.cc:460] Webserver started at http://127.12.8.193:37455/ using document root <none> and password file <none>
I20260812 02:30:54.001587 25051 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:54.001719 25051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:54.002054 25051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:54.005465 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "6e41147f7ee549a19710930bfabad79a"
format_stamp: "Formatted at 2026-08-12 02:30:53 on dist-test-slave-vq2q"
server_key: "4866e1fe1d3c4593363f60de889b67b7"
server_key_iv: "2e1275d42387c76c5d3a15ea0394eab0"
server_key_version: "encryptionkey@0"
I20260812 02:30:54.006340 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "6e41147f7ee549a19710930bfabad79a"
format_stamp: "Formatted at 2026-08-12 02:30:53 on dist-test-slave-vq2q"
server_key: "4866e1fe1d3c4593363f60de889b67b7"
server_key_iv: "2e1275d42387c76c5d3a15ea0394eab0"
server_key_version: "encryptionkey@0"
I20260812 02:30:54.013006 25051 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 02:30:54.017458 25069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:54.019363 25051 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:30:54.019579 25051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "6e41147f7ee549a19710930bfabad79a"
format_stamp: "Formatted at 2026-08-12 02:30:53 on dist-test-slave-vq2q"
server_key: "4866e1fe1d3c4593363f60de889b67b7"
server_key_iv: "2e1275d42387c76c5d3a15ea0394eab0"
server_key_version: "encryptionkey@0"
I20260812 02:30:54.019804 25051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:54.047374 25051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:54.052081 25051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:54.052397 25051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:54.053601 25051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:54.055534 25051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:54.055624 25051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:54.055751 25051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:54.055806 25051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:54.094758 25051 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:33877
I20260812 02:30:54.094904 25212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:33877 every 8 connection(s)
I20260812 02:30:54.096187 25051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:30:54.099586 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 25051
I20260812 02:30:54.099840 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:30:54.100675 12323 external_mini_cluster.cc:1428] Setting key 624ccbd437166fb91c154af4a2b14d9d
Aug 12 02:30:54 dist-test-slave-vq2q krb5kdc[24925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501853, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:54.124202 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:54.099623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:53237 (local address 127.12.8.254:33497)
0812 02:30:54.100302 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:54.100310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:54.101539 (+ 1229us) server_negotiation.cc:407] Connection header received
0812 02:30:54.104177 (+ 2638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:54.104181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:54.104256 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:54.104375 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:54.106972 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:54.107744 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:54.110140 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:54.110594 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:54.110703 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:54.116013 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:54.116022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:54.116027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:54.116083 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:54.119864 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:54.121039 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:54.121047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:54.121052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:54.121314 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:54.121810 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:54.121815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:54.121819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:54.122285 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:54.122463 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:54.123339 (+ 876us) server_negotiation.cc:299] Negotiation successful
0812 02:30:54.123668 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":317,"threads_started":1}
I20260812 02:30:54.125026 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:54.100248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33497 (local address 127.12.8.193:53237)
0812 02:30:54.101292 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:54.101360 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:54.103776 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:54.104720 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:54.104733 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:54.105705 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:54.106741 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:54.106768 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:54.107931 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:54.107936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:54.109925 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:54.109941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:54.110862 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:54.110869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:54.111185 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:54.112177 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:54.112212 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:54.115631 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:54.120108 (+ 4477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:54.120118 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:54.120137 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:54.120832 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:54.121438 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:54.121441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:54.121443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:54.121663 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:54.122441 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:54.122448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:54.122845 (+ 397us) client_negotiation.cc:769] Sending connection context
0812 02:30:54.123127 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 02:30:54.123599 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":407,"threads_started":1}
I20260812 02:30:54.126816 25213 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33497
I20260812 02:30:54.127344 25213 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:54.128546 25213 heartbeater.cc:511] Master 127.12.8.254:33497 requested a full tablet report, sending...
I20260812 02:30:54.131403 24976 ts_manager.cc:194] Registered new tserver with Master: 6e41147f7ee549a19710930bfabad79a (127.12.8.193:33877)
I20260812 02:30:54.132972 24976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:53237
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:30:54.186364 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33497
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:30:54.616068 25221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:54.616513 25221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:54.616675 25221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:54.625713 25221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:54.625886 25221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:54.626065 25221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:30:54.637988 25221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:54.640156 25221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:54.642558 25221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:54.661547 25226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:54.661593 25229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:54.661702 25227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:54.663389 25221 server_base.cc:1034] running on GCE node
I20260812 02:30:54.664289 25221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:54.665745 25221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:54.667035 25221 hybrid_clock.cc:648] HybridClock initialized: now 1786501854666963 us; error 73 us; skew 500 ppm
I20260812 02:30:54.667500 25221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:54 dist-test-slave-vq2q krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501854, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:54.672091 25221 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:30:54.674543 25221 webserver.cc:460] Webserver started at http://127.12.8.194:43029/ using document root <none> and password file <none>
I20260812 02:30:54.675604 25221 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:54.675781 25221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:54.676206 25221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:54.680294 25221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "2f51b37002674f8fa0f6ff06163649b9"
format_stamp: "Formatted at 2026-08-12 02:30:54 on dist-test-slave-vq2q"
server_key: "028d770cdb986ddf7358733d8cabf8c7"
server_key_iv: "9429a6633952d1da3c61b06f05891a44"
server_key_version: "encryptionkey@0"
I20260812 02:30:54.681399 25221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "2f51b37002674f8fa0f6ff06163649b9"
format_stamp: "Formatted at 2026-08-12 02:30:54 on dist-test-slave-vq2q"
server_key: "028d770cdb986ddf7358733d8cabf8c7"
server_key_iv: "9429a6633952d1da3c61b06f05891a44"
server_key_version: "encryptionkey@0"
I20260812 02:30:54.688858 25221 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 02:30:54.693939 25241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:54.696456 25221 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 02:30:54.696818 25221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "2f51b37002674f8fa0f6ff06163649b9"
format_stamp: "Formatted at 2026-08-12 02:30:54 on dist-test-slave-vq2q"
server_key: "028d770cdb986ddf7358733d8cabf8c7"
server_key_iv: "9429a6633952d1da3c61b06f05891a44"
server_key_version: "encryptionkey@0"
I20260812 02:30:54.697140 25221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:54.713984 25221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:54.720539 25221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:54.720996 25221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:54.722450 25221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:54.724632 25221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:54.724748 25221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:54.724908 25221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:54.724973 25221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:54.768148 25221 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:34971
I20260812 02:30:54.768188 25380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:34971 every 8 connection(s)
I20260812 02:30:54.769974 25221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:30:54.776049 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 25221
I20260812 02:30:54.776276 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:30:54.776753 12323 external_mini_cluster.cc:1428] Setting key 28a75d26f1b247f559725917a681d2ed
Aug 12 02:30:54 dist-test-slave-vq2q krb5kdc[24925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501854, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:54.795528 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:54.773107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:53889 (local address 127.12.8.254:33497)
0812 02:30:54.773723 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:54.773730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:54.774881 (+ 1151us) server_negotiation.cc:407] Connection header received
0812 02:30:54.777428 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:54.777436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:54.777550 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:54.777673 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:54.779786 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:54.780585 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:54.782495 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:54.782962 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:54.783085 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:54.788162 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:54.788169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:54.788173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:54.788222 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:54.791362 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:54.792398 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:54.792403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:54.792405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:54.792557 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:54.793140 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:54.793144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:54.793146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:54.793680 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:54.793853 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:54.794821 (+ 968us) server_negotiation.cc:299] Negotiation successful
0812 02:30:54.795120 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":310,"threads_started":1}
I20260812 02:30:54.796887 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:54.773529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33497 (local address 127.12.8.194:53889)
0812 02:30:54.774648 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:54.774709 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:54.776969 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:54.778019 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:54.778030 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:54.778739 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:54.779548 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:54.779573 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:54.780768 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:54.780773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:54.782313 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:54.782337 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:54.783240 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:54.783249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:54.783540 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:54.784456 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:54.784496 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:54.787797 (+ 3301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:54.791615 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:54.791625 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:54.791647 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:54.792241 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:54.792676 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:54.792680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:54.792683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:54.792969 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:54.793834 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:54.793844 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:54.794357 (+ 513us) client_negotiation.cc:769] Sending connection context
0812 02:30:54.794675 (+ 318us) client_negotiation.cc:240] Negotiation successful
0812 02:30:54.795417 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":475,"threads_started":1}
I20260812 02:30:54.799121 25381 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33497
I20260812 02:30:54.799569 25381 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:54.800531 25381 heartbeater.cc:511] Master 127.12.8.254:33497 requested a full tablet report, sending...
I20260812 02:30:54.803004 24976 ts_manager.cc:194] Registered new tserver with Master: 2f51b37002674f8fa0f6ff06163649b9 (127.12.8.194:34971)
I20260812 02:30:54.804306 24976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:53889
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:30:54.853009 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33497
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:30:55.137488 25213 heartbeater.cc:503] Master 127.12.8.254:33497 was elected leader, sending a full tablet report...
W20260812 02:30:55.223351 25388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:30:55.223691 25388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:30:55.223852 25388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:30:55.231292 25388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:30:55.231406 25388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:30:55.231518 25388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:30:55.241495 25388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:30:55.242853 25388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:30:55.244410 25388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:30:55.259109 25398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:30:55.259195 25395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:55.259687 25388 server_base.cc:1034] running on GCE node
W20260812 02:30:55.259865 25396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:30:55.260525 25388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:30:55.261565 25388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:30:55.262815 25388 hybrid_clock.cc:648] HybridClock initialized: now 1786501855262765 us; error 38 us; skew 500 ppm
I20260812 02:30:55.263151 25388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:30:55 dist-test-slave-vq2q krb5kdc[24925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501855, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:30:55.266575 25388 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:30:55.268113 25388 webserver.cc:460] Webserver started at http://127.12.8.195:44565/ using document root <none> and password file <none>
I20260812 02:30:55.268827 25388 fs_manager.cc:359] Metadata directory not provided
I20260812 02:30:55.268946 25388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:30:55.269308 25388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:30:55.272373 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "a987bbc094b34d398e091f6d597e5830"
format_stamp: "Formatted at 2026-08-12 02:30:55 on dist-test-slave-vq2q"
server_key: "70735d73eacff064222a624d8aa7fef8"
server_key_iv: "4ad186d999e073660ae8bfd25fbe307c"
server_key_version: "encryptionkey@0"
I20260812 02:30:55.273125 25388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "a987bbc094b34d398e091f6d597e5830"
format_stamp: "Formatted at 2026-08-12 02:30:55 on dist-test-slave-vq2q"
server_key: "70735d73eacff064222a624d8aa7fef8"
server_key_iv: "4ad186d999e073660ae8bfd25fbe307c"
server_key_version: "encryptionkey@0"
I20260812 02:30:55.278394 25388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 02:30:55.282001 25406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:55.283440 25388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:30:55.283610 25388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "a987bbc094b34d398e091f6d597e5830"
format_stamp: "Formatted at 2026-08-12 02:30:55 on dist-test-slave-vq2q"
server_key: "70735d73eacff064222a624d8aa7fef8"
server_key_iv: "4ad186d999e073660ae8bfd25fbe307c"
server_key_version: "encryptionkey@0"
I20260812 02:30:55.283800 25388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:30:55.299501 25388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:30:55.305328 25388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:30:55.305624 25388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:30:55.306684 25388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:30:55.308115 25388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:30:55.308193 25388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:55.308313 25388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:30:55.308369 25388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:30:55.340086 25388 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:38443
I20260812 02:30:55.340132 25536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:38443 every 8 connection(s)
I20260812 02:30:55.341444 25388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:30:55.351281 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 25388
I20260812 02:30:55.351547 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:30:55.352088 12323 external_mini_cluster.cc:1428] Setting key 5a597759c0e5da4e08004867a08dd4d2
Aug 12 02:30:55 dist-test-slave-vq2q krb5kdc[24925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501855, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:30:55.361864 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.344053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:53543 (local address 127.12.8.254:33497)
0812 02:30:55.344537 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.344542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.345475 (+ 933us) server_negotiation.cc:407] Connection header received
0812 02:30:55.347279 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.347283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.347355 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.347459 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:55.348879 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.349753 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.351252 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.351817 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.351955 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.356066 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:55.356073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:55.356079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:55.356113 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:55.358753 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:55.359508 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:55.359514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:55.359517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:55.359642 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:55.360057 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:55.360062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:55.360066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:55.360348 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:55.360503 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.361237 (+ 734us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.361474 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
I20260812 02:30:55.362830 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.344549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33497 (local address 127.12.8.195:53543)
0812 02:30:55.345314 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:55.345372 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:55.347041 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:55.347647 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:55.347654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:55.348149 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:30:55.348724 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.348743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.349901 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.349905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.351107 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.351119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.352032 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.352037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.352265 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.353029 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:30:55.353059 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:30:55.355854 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:30:55.358890 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:55.358899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:55.358915 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:55.359401 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:55.359744 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:30:55.359749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:30:55.359753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:30:55.359952 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:30:55.360481 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:30:55.360488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:30:55.360844 (+ 356us) client_negotiation.cc:769] Sending connection context
0812 02:30:55.361098 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 02:30:55.361658 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":272,"threads_started":1}
I20260812 02:30:55.364094 25537 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33497
I20260812 02:30:55.364353 25537 heartbeater.cc:464] Registering TS with master...
I20260812 02:30:55.365051 25537 heartbeater.cc:511] Master 127.12.8.254:33497 requested a full tablet report, sending...
I20260812 02:30:55.366652 24976 ts_manager.cc:194] Registered new tserver with Master: a987bbc094b34d398e091f6d597e5830 (127.12.8.195:38443)
I20260812 02:30:55.367630 24976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:53543
I20260812 02:30:55.367892 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:30:55.385546 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.373403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.12.8.254:33497)
0812 02:30:55.373584 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.373590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.374005 (+ 415us) server_negotiation.cc:407] Connection header received
0812 02:30:55.374162 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.374166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.374228 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.374310 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:55.375719 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.376435 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.377695 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.378107 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.378223 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.380461 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:55.380466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:55.380469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:55.380500 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:55.382982 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:55.383611 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:55.383615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:55.383617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:55.383768 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:55.384102 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:55.384106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:55.384108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:55.384443 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:55.384677 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.384997 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.385248 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 02:30:55.389741 24976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:30:55.396606 24976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:30:55.423347 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.409728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.12.8.195:38443)
0812 02:30:55.411609 (+ 1881us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.411617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.411652 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:30:55.412832 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.412843 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.413127 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.413378 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:55.414952 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.416787 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.419875 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.421501 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.421613 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.421880 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.422671 (+ 791us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.422869 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1707,"thread_start_us":1722,"threads_started":1}
I20260812 02:30:55.423830 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.409101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:38443 (local address 127.0.0.1:33900)
0812 02:30:55.410891 (+ 1790us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:55.410910 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:55.411031 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:55.414140 (+ 3109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:55.414149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:55.414185 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:55.414659 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.414677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.417002 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.417008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.419653 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.419670 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.422097 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.422103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.422353 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.422414 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 02:30:55.423274 (+ 860us) client_negotiation.cc:240] Negotiation successful
0812 02:30:55.423489 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":1510,"threads_started":1}
I20260812 02:30:55.425598 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.408944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:33877 (local address 127.0.0.1:55214)
0812 02:30:55.410214 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:55.410265 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:55.410431 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:55.414755 (+ 4324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:55.414761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:55.414790 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:55.415283 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.415295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.418823 (+ 3528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.418829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.421203 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.421221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.424781 (+ 3560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.424788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.424970 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.425003 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 02:30:55.425068 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 02:30:55.425217 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":882,"threads_started":1}
I20260812 02:30:55.426195 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.409334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.12.8.193:33877)
0812 02:30:55.414165 (+ 4831us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.414175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.414204 (+ 29us) server_negotiation.cc:407] Connection header received
0812 02:30:55.414303 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.414310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.414531 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.416594 (+ 2063us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:55.416885 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.418646 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.422707 (+ 4061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.424570 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.425371 (+ 801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.425612 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.425684 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.425877 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4638,"thread_start_us":4444,"threads_started":1}
I20260812 02:30:55.428035 25464 tablet_service.cc:1467] Processing CreateTablet for tablet 64bff8ceba734d4fabc369a5ec634fed (DEFAULT_TABLE table=test-table [id=d012502312e04131a26348f180de13ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:55.429675 25464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bff8ceba734d4fabc369a5ec634fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:55.429677 25132 tablet_service.cc:1467] Processing CreateTablet for tablet 64bff8ceba734d4fabc369a5ec634fed (DEFAULT_TABLE table=test-table [id=d012502312e04131a26348f180de13ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:55.431036 25132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bff8ceba734d4fabc369a5ec634fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:55.431962 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.411338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:34971 (local address 127.0.0.1:46800)
0812 02:30:55.422870 (+ 11532us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:55.422890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:55.423000 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:55.425074 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:55.425079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:55.425104 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:55.425527 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.425540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.428420 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.428424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.429813 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.429827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.431212 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.431222 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.431366 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.431394 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:30:55.431465 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 02:30:55.431592 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11403,"thread_start_us":11740,"threads_started":1}
I20260812 02:30:55.432024 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.411671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.12.8.194:34971)
0812 02:30:55.424585 (+ 12914us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.424594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.424622 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:30:55.424706 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.424711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.424893 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.425062 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:55.426110 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.428276 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.429963 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.430940 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.431073 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.431257 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.431543 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.431712 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12753,"thread_start_us":12621,"threads_started":1}
I20260812 02:30:55.436255 25301 tablet_service.cc:1467] Processing CreateTablet for tablet 64bff8ceba734d4fabc369a5ec634fed (DEFAULT_TABLE table=test-table [id=d012502312e04131a26348f180de13ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:30:55.437635 25301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64bff8ceba734d4fabc369a5ec634fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:30:55.444432 25563 tablet_bootstrap.cc:492] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Bootstrap starting.
I20260812 02:30:55.446508 25564 tablet_bootstrap.cc:492] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Bootstrap starting.
I20260812 02:30:55.448048 25563 tablet_bootstrap.cc:654] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:55.449730 25563 log.cc:824] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:55.449935 25564 tablet_bootstrap.cc:654] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:55.451128 25564 log.cc:824] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:55.452279 25568 tablet_bootstrap.cc:492] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Bootstrap starting.
I20260812 02:30:55.452731 25563 tablet_bootstrap.cc:492] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: No bootstrap required, opened a new log
I20260812 02:30:55.453058 25563 ts_tablet_manager.cc:1397] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 02:30:55.455202 25564 tablet_bootstrap.cc:492] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: No bootstrap required, opened a new log
I20260812 02:30:55.455466 25564 ts_tablet_manager.cc:1397] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 02:30:55.456008 25568 tablet_bootstrap.cc:654] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Neither blocks nor log segments found. Creating new log.
I20260812 02:30:55.457280 25568 log.cc:824] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:30:55.460953 25568 tablet_bootstrap.cc:492] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: No bootstrap required, opened a new log
I20260812 02:30:55.461238 25568 ts_tablet_manager.cc:1397] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 02:30:55.463093 25563 raft_consensus.cc:348] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.463243 25564 raft_consensus.cc:348] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.463562 25563 raft_consensus.cc:374] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:55.463581 25564 raft_consensus.cc:374] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:55.463665 25564 raft_consensus.cc:729] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e41147f7ee549a19710930bfabad79a, State: Initialized, Role: FOLLOWER
I20260812 02:30:55.463665 25563 raft_consensus.cc:729] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a987bbc094b34d398e091f6d597e5830, State: Initialized, Role: FOLLOWER
I20260812 02:30:55.464224 25564 consensus_queue.cc:260] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.464329 25563 consensus_queue.cc:260] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.465548 25537 heartbeater.cc:503] Master 127.12.8.254:33497 was elected leader, sending a full tablet report...
I20260812 02:30:55.466187 25563 ts_tablet_manager.cc:1428] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260812 02:30:55.467631 25564 ts_tablet_manager.cc:1428] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260812 02:30:55.471871 25568 raft_consensus.cc:348] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.472295 25568 raft_consensus.cc:374] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:30:55.472410 25568 raft_consensus.cc:729] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f51b37002674f8fa0f6ff06163649b9, State: Initialized, Role: FOLLOWER
I20260812 02:30:55.472975 25568 consensus_queue.cc:260] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.474152 25381 heartbeater.cc:503] Master 127.12.8.254:33497 was elected leader, sending a full tablet report...
I20260812 02:30:55.474691 25568 ts_tablet_manager.cc:1428] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260812 02:30:55.522413 25382 tablet.cc:2369] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:30:55.593223 25538 tablet.cc:2369] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:30:55.599092 25216 tablet.cc:2369] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:30:55.670727 25575 raft_consensus.cc:482] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:30:55.671029 25575 raft_consensus.cc:504] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.672775 25575 leader_election.cc:288] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e41147f7ee549a19710930bfabad79a (127.12.8.193:33877), a987bbc094b34d398e091f6d597e5830 (127.12.8.195:38443)
I20260812 02:30:55.679177 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.673221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:46439 (local address 127.12.8.193:33877)
0812 02:30:55.673444 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.673450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.674086 (+ 636us) server_negotiation.cc:407] Connection header received
0812 02:30:55.674299 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.674303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.674365 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.674448 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:55.674957 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.675677 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.677522 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.678302 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.678463 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.678542 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.678796 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.678925 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:30:55.679280 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.673231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:33877 (local address 127.12.8.194:46439)
0812 02:30:55.673974 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:55.673990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:55.674158 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:55.674477 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:55.674482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:55.674507 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:55.674818 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.674828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.675801 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.675804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.677365 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.677379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.678507 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.678514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.678650 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.678674 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:30:55.678762 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:30:55.678912 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":400,"threads_started":1}
I20260812 02:30:55.679489 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.673386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:36867 (local address 127.12.8.195:38443)
0812 02:30:55.673646 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.673652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.674319 (+ 667us) server_negotiation.cc:407] Connection header received
0812 02:30:55.674513 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.674517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.674587 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.674664 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:30:55.675261 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.676035 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.677809 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.678598 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.678758 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.678844 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.679127 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.679252 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:30:55.679611 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.673253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:38443 (local address 127.12.8.194:36867)
0812 02:30:55.674212 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0812 02:30:55.674226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:30:55.674378 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:30:55.674700 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:30:55.674704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:30:55.674732 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:30:55.675113 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.675122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.676170 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.676176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.677653 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:30:55.677666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.678819 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.678825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:30:55.678975 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.679003 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:30:55.679086 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:30:55.679212 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":4,"thread_start_us":608,"threads_started":1}
I20260812 02:30:55.680540 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bff8ceba734d4fabc369a5ec634fed" candidate_uuid: "2f51b37002674f8fa0f6ff06163649b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a987bbc094b34d398e091f6d597e5830" is_pre_election: true
I20260812 02:30:55.680523 25157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bff8ceba734d4fabc369a5ec634fed" candidate_uuid: "2f51b37002674f8fa0f6ff06163649b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e41147f7ee549a19710930bfabad79a" is_pre_election: true
I20260812 02:30:55.681123 25486 raft_consensus.cc:2454] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f51b37002674f8fa0f6ff06163649b9 in term 0.
I20260812 02:30:55.681121 25157 raft_consensus.cc:2454] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f51b37002674f8fa0f6ff06163649b9 in term 0.
I20260812 02:30:55.681865 25245 leader_election.cc:302] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f51b37002674f8fa0f6ff06163649b9, a987bbc094b34d398e091f6d597e5830; no voters:
I20260812 02:30:55.682242 25575 raft_consensus.cc:2781] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:30:55.682385 25575 raft_consensus.cc:482] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:30:55.682489 25575 raft_consensus.cc:3037] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:55.684724 25575 raft_consensus.cc:504] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.685410 25575 leader_election.cc:288] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [CANDIDATE]: Term 1 election: Requested vote from peers 6e41147f7ee549a19710930bfabad79a (127.12.8.193:33877), a987bbc094b34d398e091f6d597e5830 (127.12.8.195:38443)
I20260812 02:30:55.685804 25157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bff8ceba734d4fabc369a5ec634fed" candidate_uuid: "2f51b37002674f8fa0f6ff06163649b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e41147f7ee549a19710930bfabad79a"
I20260812 02:30:55.685899 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64bff8ceba734d4fabc369a5ec634fed" candidate_uuid: "2f51b37002674f8fa0f6ff06163649b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a987bbc094b34d398e091f6d597e5830"
I20260812 02:30:55.686007 25157 raft_consensus.cc:3037] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:55.686095 25486 raft_consensus.cc:3037] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:30:55.688309 25486 raft_consensus.cc:2454] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f51b37002674f8fa0f6ff06163649b9 in term 1.
I20260812 02:30:55.688366 25157 raft_consensus.cc:2454] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f51b37002674f8fa0f6ff06163649b9 in term 1.
I20260812 02:30:55.688669 25245 leader_election.cc:302] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f51b37002674f8fa0f6ff06163649b9, a987bbc094b34d398e091f6d597e5830; no voters:
I20260812 02:30:55.688970 25575 raft_consensus.cc:2781] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:30:55.689481 25575 raft_consensus.cc:686] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f51b37002674f8fa0f6ff06163649b9, State: Running, Role: LEADER
I20260812 02:30:55.689971 25575 consensus_queue.cc:237] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } }
I20260812 02:30:55.694837 24976 catalog_manager.cc:5495] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f51b37002674f8fa0f6ff06163649b9 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "2f51b37002674f8fa0f6ff06163649b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f51b37002674f8fa0f6ff06163649b9" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 34971 } health_report { overall_health: HEALTHY } } }
I20260812 02:30:55.715384 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.710361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.12.8.194:34971)
0812 02:30:55.710579 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.710584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.710602 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:30:55.710782 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.710785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.710864 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.711009 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:30:55.711320 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.712042 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.713403 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.713800 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.713895 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.714306 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:30:55.714799 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:30:55.714924 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.715035 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.715101 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:30:55.726512 25157 raft_consensus.cc:1261] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 1 FOLLOWER]: Refusing update from remote peer 2f51b37002674f8fa0f6ff06163649b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:55.726512 25486 raft_consensus.cc:1261] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 1 FOLLOWER]: Refusing update from remote peer 2f51b37002674f8fa0f6ff06163649b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:30:55.727644 25575 consensus_queue.cc:1035] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e41147f7ee549a19710930bfabad79a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:55.727957 25578 consensus_queue.cc:1035] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a987bbc094b34d398e091f6d597e5830" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:30:55.743335 25581 mvcc.cc:205] Tried to move back new op lower bound from 7317511601042632704 to 7317511600909987840. Current Snapshot: MvccSnapshot[applied={T|T < 7317511601042632704}]
I20260812 02:30:55.756388 24976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:55.756642 24976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:30:55.760388 24976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04c5ab9b062249ac94890599ca0663e5: Sending DeleteTablet for 3 replicas of tablet 64bff8ceba734d4fabc369a5ec634fed
I20260812 02:30:55.761547 25132 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bff8ceba734d4fabc369a5ec634fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:55 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:55214
I20260812 02:30:55.761806 25301 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bff8ceba734d4fabc369a5ec634fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:55 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:46800
I20260812 02:30:55.761871 25464 tablet_service.cc:1514] Processing DeleteTablet for tablet 64bff8ceba734d4fabc369a5ec634fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:30:55 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33900
I20260812 02:30:55.763916 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 25051
I20260812 02:30:55.800921 25593 tablet_replica.cc:330] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: stopping tablet replica
I20260812 02:30:55.803398 25591 tablet_replica.cc:330] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: stopping tablet replica
I20260812 02:30:55.807376 25591 raft_consensus.cc:2229] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:55.805780 25593 raft_consensus.cc:2229] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:30:55.808255 25593 raft_consensus.cc:2258] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:55.808872 25592 tablet_replica.cc:330] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: stopping tablet replica
I20260812 02:30:55.809224 25592 raft_consensus.cc:2229] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:30:55.810005 25591 raft_consensus.cc:2258] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:55.810051 25592 raft_consensus.cc:2258] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:30:55.813087 25593 ts_tablet_manager.cc:1905] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:55.813539 25591 ts_tablet_manager.cc:1905] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:55.815018 25592 ts_tablet_manager.cc:1905] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:30:55.820711 25591 ts_tablet_manager.cc:1918] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:55.820907 25591 log.cc:1196] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/64bff8ceba734d4fabc369a5ec634fed
I20260812 02:30:55.821513 25591 ts_tablet_manager.cc:1939] T 64bff8ceba734d4fabc369a5ec634fed P 6e41147f7ee549a19710930bfabad79a: Deleting consensus metadata
Aug 12 02:30:55 dist-test-slave-vq2q krb5kdc[24925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:30:55.826912 24961 catalog_manager.cc:4841] TS 6e41147f7ee549a19710930bfabad79a (127.12.8.193:33877): tablet 64bff8ceba734d4fabc369a5ec634fed (table test-table [id=d012502312e04131a26348f180de13ef]) successfully deleted
I20260812 02:30:55.831103 25593 ts_tablet_manager.cc:1918] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:55.831398 25593 log.cc:1196] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/64bff8ceba734d4fabc369a5ec634fed
I20260812 02:30:55.832082 25593 ts_tablet_manager.cc:1939] T 64bff8ceba734d4fabc369a5ec634fed P a987bbc094b34d398e091f6d597e5830: Deleting consensus metadata
I20260812 02:30:55.834838 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:55.765512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.12.8.193:33877)
0812 02:30:55.765702 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:55.765709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:55.797903 (+ 32194us) server_negotiation.cc:407] Connection header received
0812 02:30:55.798242 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:55.798249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:55.798375 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:55.798457 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:55.801254 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.802597 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.810823 (+ 8226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:55.811421 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:55.811596 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:55.826665 (+ 15069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:55.826683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:55.826701 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:55.826759 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:55.830475 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:55.831528 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:55.831537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:55.831540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:55.831717 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:55.832187 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:55.832192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:55.832195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:55.832703 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:55.833014 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:55.834214 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0812 02:30:55.834458 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:30:55.835712 25207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:55.835908 25592 ts_tablet_manager.cc:1918] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:30:55.836189 25592 log.cc:1196] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/64bff8ceba734d4fabc369a5ec634fed
I20260812 02:30:55.837044 25592 ts_tablet_manager.cc:1939] T 64bff8ceba734d4fabc369a5ec634fed P 2f51b37002674f8fa0f6ff06163649b9: Deleting consensus metadata
I20260812 02:30:55.840291 24960 catalog_manager.cc:4841] TS a987bbc094b34d398e091f6d597e5830 (127.12.8.195:38443): tablet 64bff8ceba734d4fabc369a5ec634fed (table test-table [id=d012502312e04131a26348f180de13ef]) successfully deleted
I20260812 02:30:55.841332 24960 catalog_manager.cc:4841] TS 2f51b37002674f8fa0f6ff06163649b9 (127.12.8.194:34971): tablet 64bff8ceba734d4fabc369a5ec634fed (table test-table [id=d012502312e04131a26348f180de13ef]) successfully deleted
I20260812 02:30:55.990393 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 25051
I20260812 02:30:56.011842 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 25221
Aug 12 02:30:56 dist-test-slave-vq2q krb5kdc[24925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:30:56.028221 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:56.012594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.12.8.194:34971)
0812 02:30:56.012783 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:56.012789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:56.012911 (+ 122us) server_negotiation.cc:407] Connection header received
0812 02:30:56.013097 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:56.013103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:56.013210 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:56.013321 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:56.014555 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:56.015398 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:56.016616 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:56.017082 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:56.017372 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:56.021588 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:56.021603 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:56.021618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:56.021668 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:56.025006 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:56.025974 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:56.025983 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:56.025985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:56.026159 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:56.026554 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:56.026558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:56.026560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:56.026963 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:56.027334 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:56.027616 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 02:30:56.027866 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:30:56.028956 25375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:56.163668 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 25221
I20260812 02:30:56.188683 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 25388
Aug 12 02:30:56 dist-test-slave-vq2q krb5kdc[24925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:30:56.206223 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:30:56.189605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.12.8.195:38443)
0812 02:30:56.190086 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 02:30:56.190091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:30:56.190112 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:30:56.190177 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:30:56.190181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:30:56.190247 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:30:56.190358 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:30:56.192072 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:56.192796 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:56.194239 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:30:56.194639 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:30:56.194740 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:30:56.199756 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:30:56.199778 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:30:56.199796 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:30:56.199839 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:30:56.202874 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:56.203942 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:56.203950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:56.203953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:56.204146 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:30:56.204692 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:30:56.204695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:30:56.204697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:30:56.205096 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:30:56.205404 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 02:30:56.205719 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 02:30:56.205922 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":283,"threads_started":1}
I20260812 02:30:56.207127 25531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:56.298728 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 25388
I20260812 02:30:56.326960 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 24941
I20260812 02:30:56.328414 25012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:30:56.444155 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 24941
2026-08-12T02:30:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5185 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:30:56 dist-test-slave-vq2q krb5kdc[25603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:30:56 dist-test-slave-vq2q krb5kdc[25603](info): set up 2 sockets
Aug 12 02:30:56 dist-test-slave-vq2q krb5kdc[25603](info): commencing operation
krb5kdc: starting...
W20260812 02:30:57.630510 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.001s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:30:57 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:30:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:30:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.8.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.8.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/rangeradmin_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/rangeradmin_127.12.8.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.8.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.8.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/rangerlookup_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/rangerlookup_127.12.8.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.8.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.8.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/HTTP_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/HTTP_127.12.8.212@KRBTEST.COM.keytab.
I20260812 02:30:58.173417 12323 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqIRTh5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/postgres -l logfile start
2026-08-12 02:31:00.782 UTC [25643] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 02:31:00.782 UTC [25643] LOG: listening on IPv4 address "127.12.8.212", port 44145
2026-08-12 02:31:00.787 UTC [25643] LOG: listening on Unix socket "/tmp/.s.PGSQL.44145"
2026-08-12 02:31:00.816 UTC [25645] LOG: database system was shut down at 2026-08-12 02:30:59 UTC
2026-08-12 02:31:00.819 UTC [25643] LOG: database system is ready to accept connections
I20260812 02:31:01.822427 12323 mini_postgres.cc:96] Postgres bound to 44145
2026-08-12 02:31:01.848 UTC [25656] FATAL: database "slave" does not exist
127.12.8.212:44145 - accepting connections
I20260812 02:31:01.849738 12323 mini_ranger.cc:161] Starting Ranger...
I20260812 02:31:01.884991 12323 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 02:31:02.397291 12323 mini_ranger.cc:87] Created ranger Postgres database
I20260812 02:31:02.397472 12323 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-admin
2026-08-12 02:31:02,728 [I] DB FLAVOR :POSTGRES
2026-08-12 02:31:02,728 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:31:02,728 [I] Checking connection..
2026-08-12 02:31:02,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 02:31:03,009 [I] Checking connection passed.
2026-08-12 02:31:03,010 [I] --------- Verifying version history table ---------
2026-08-12 02:31:03,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:31:03,266 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:31:03,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:31:03,519 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:31:03,520 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 02:31:03,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 02:31:03.750 UTC [25736] WARNING: there is no transaction in progress
2026-08-12 02:31:03,758 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 02:31:03,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:31:03,998 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 02:31:03,998 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:31:03,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 02:31:04,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 02:31:04,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vq2q.c.gcp-upstream.internal','N') ;"
2026-08-12 02:31:04,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:31:05,013 [I] Table x_portal_user does not exist in database ranger
2026-08-12 02:31:05,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:31:05,285 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 02:31:05,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:31:05,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:31:05,808 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 02:31:05,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 02:31:06.955 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:06.975 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:06.994 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.015 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.035 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.055 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.118 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.126 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.144 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.157 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.169 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.185 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.197 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.204 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.212 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07.220 UTC [25867] WARNING: there is no transaction in progress
2026-08-12 02:31:07,977 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 02:31:07,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 02:31:08,244 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 02:31:08,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:31:08,532 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 02:31:08,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:31:08,811 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 02:31:08,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:31:09,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:31:09,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vq2q.c.gcp-upstream.internal';"
2026-08-12 02:31:09,657 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 02:31:09,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 02:31:09,931 [I] DB_PATCHES have already been applied
I20260812 02:31:09.939323 12323 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqIRTh5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 02:31:09.939435 12323 mini_ranger.cc:193] Using host: 127.12.8.212
I20260812 02:31:09.967392 12323 mini_ranger.cc:227] Ranger admin URL: http://127.12.8.212:34817
Aug 12, 2026 2:31:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:31:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:31:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:31:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:31:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:31:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:31:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 02:31:11.165549 12323 mini_ranger.cc:160] Time spent starting Ranger: real 9.316s user 0.013s sys 0.075s
Aug 12, 2026 2:31:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:31:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 12, 2026 2:31:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:31:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:31:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 02:31:37.071851 12323 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.8.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.8.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/rangerkms_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/rangerkms_127.12.8.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.8.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.8.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.8.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/HTTP_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.8.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/HTTP_127.12.8.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 02:31:37 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501897, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 02:31:37.300240 12323 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 02:31:37.330602 12323 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 02:31:38.180317 12323 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 02:31:38.180533 12323 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms
I20260812 02:31:38.180677 12323 mini_ranger_kms.cc:227] Using postgres at 127.12.8.212:44145
2026-08-12 02:31:38,916 [I] DB FLAVOR :POSTGRES
2026-08-12 02:31:38,916 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:31:38,916 [I] Checking connection
2026-08-12 02:31:38,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 02:31:39,178 [I] connection success
2026-08-12 02:31:39,179 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 02:31:39,179 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 02:31:39,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 02:31:39,438 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 02:31:39,438 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:31:39,438 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 02:31:39,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:44145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 02:31:39,721 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 02:31:40.239992 12323 mini_ranger_kms.cc:310] Created kms service
I20260812 02:31:40.459647 12323 mini_ranger_kms.cc:326] Created kudu user
I20260812 02:31:40.506011 12323 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 02:31:40 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501897, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:31:40.952026 12323 mini_ranger_kms.cc:384] Added ranger policy
I20260812 02:31:40.953833 12323 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 02:31:40.953922 12323 mini_ranger_kms.cc:240] Using host: 127.12.8.212
I20260812 02:31:40.985147 12323 mini_ranger_kms.cc:276] Ranger KMS PID: 26087
I20260812 02:31:40.985419 12323 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.8.212:35915
Aug 12, 2026 2:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:31:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 02:31:43 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786501903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:43.434401 12323 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.134s user 0.178s sys 0.620s
I20260812 02:31:43.434790 12323 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 02:31:43.435024 12323 mini_ranger_kms.cc:401] 127.12.8.212:35915/kms/v1/keys
Aug 12 02:31:43 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.212@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
Aug 12 02:31:44 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501903, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.212@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
Aug 12 02:31:46 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:31:46.882187 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40897
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40897
--ranger_config_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:35915/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:47.603928 26146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:47.604359 26146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:47.604573 26146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:47.615897 26146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:31:47.616140 26146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:47.616258 26146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:47.616418 26146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:31:47.616559 26146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:31:47.632670 26146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.8.212:35915/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40897
--ranger_config_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:31:47.634477 26146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:47.636190 26146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:47.657877 26153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:47.658069 26152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:47.658138 26155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:47.663614 26146 server_base.cc:1034] running on GCE node
I20260812 02:31:47.664710 26146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:47.666700 26146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:47.667990 26146 hybrid_clock.cc:648] HybridClock initialized: now 1786501907667920 us; error 98 us; skew 500 ppm
I20260812 02:31:47.668561 26146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:31:47 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501907, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:47.674072 26146 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:31:47.676412 26146 webserver.cc:460] Webserver started at http://127.12.8.254:36635/ using document root <none> and password file <none>
I20260812 02:31:47.677311 26146 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:47.677538 26146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:47.678018 26146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:31:47 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501907, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:31:47.749089 26146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8"
format_stamp: "Formatted at 2026-08-12 02:31:47 on dist-test-slave-vq2q"
server_key: "09dafcb5f03e4239df22aff74b2fd0dc"
server_key_iv: "4252afddacd6b5bcf439285ba9e0fb36"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:47.750545 26146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8"
format_stamp: "Formatted at 2026-08-12 02:31:47 on dist-test-slave-vq2q"
server_key: "09dafcb5f03e4239df22aff74b2fd0dc"
server_key_iv: "4252afddacd6b5bcf439285ba9e0fb36"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:47.760111 26146 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260812 02:31:47.789839 26162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:47.791992 26146 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260812 02:31:47.792305 26146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8"
format_stamp: "Formatted at 2026-08-12 02:31:47 on dist-test-slave-vq2q"
server_key: "09dafcb5f03e4239df22aff74b2fd0dc"
server_key_iv: "4252afddacd6b5bcf439285ba9e0fb36"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:47.792680 26146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:47.814136 26146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:47.821362 26146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:47.822042 26146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:47.885546 26146 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:40897
I20260812 02:31:47.885624 26228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:40897 every 8 connection(s)
I20260812 02:31:47.888036 26146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:31:47.896727 26229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.897909 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 26146
I20260812 02:31:47.898293 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:31:47.899554 12323 external_mini_cluster.cc:1428] Setting key 23f0d69fda146813f50885dd6105faf6
Aug 12 02:31:47 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:31:47.923960 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8: Bootstrap starting.
I20260812 02:31:47.931249 26229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.933681 26229 log.cc:824] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:47.939829 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8: No bootstrap required, opened a new log
I20260812 02:31:47.940933 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.903083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.12.8.254:40897)
0812 02:31:47.905633 (+ 2550us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.905652 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.905855 (+ 203us) server_negotiation.cc:407] Connection header received
0812 02:31:47.910039 (+ 4184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.910086 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.910921 (+ 835us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.911757 (+ 836us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:47.913941 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.916250 (+ 2309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.918234 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.919557 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.919854 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.925734 (+ 5880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:47.925751 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:47.925774 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:47.925905 (+ 131us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:47.933105 (+ 7200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:47.934518 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:47.934533 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:47.934546 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:47.934964 (+ 418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:47.935978 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:47.935986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:47.935989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:47.937339 (+ 1350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:47.937808 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.938344 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.938881 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2146,"thread_start_us":1303,"threads_started":1}
I20260812 02:31:47.954274 26229 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40897 } }
I20260812 02:31:47.954950 26229 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.955125 26229 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6fbdce6ec342e1b7d4a96cf7592ea8, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.955928 26229 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40897 } }
I20260812 02:31:47.956382 26229 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:31:47.956645 26229 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:31:47.956933 26229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.960875 26229 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40897 } }
I20260812 02:31:47.961540 26229 leader_election.cc:302] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf6fbdce6ec342e1b7d4a96cf7592ea8; no voters:
I20260812 02:31:47.962844 26229 leader_election.cc:288] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:31:47.963455 26234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:47.965921 26234 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [term 1 LEADER]: Becoming Leader. State: Replica: cf6fbdce6ec342e1b7d4a96cf7592ea8, State: Running, Role: LEADER
I20260812 02:31:47.966866 26229 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:31:47.967033 26234 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40897 } }
I20260812 02:31:47.976428 26235 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40897 } } }
I20260812 02:31:47.975940 26236 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf6fbdce6ec342e1b7d4a96cf7592ea8. Latest consensus state: current_term: 1 leader_uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6fbdce6ec342e1b7d4a96cf7592ea8" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40897 } } }
I20260812 02:31:47.977882 26235 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:47.977918 26236 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8 [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:47.982779 26244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:31:47.994350 26244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:31:47.996007 26229 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:31:48.025915 26244 catalog_manager.cc:1347] Generated new cluster ID: 9a5da86a6f6c4baab1303ab4381b94be
I20260812 02:31:48.026217 26244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:31:48.048607 26244 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:31:48.051411 26244 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:31:48.067025 26244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8: Generated new TSK 0
I20260812 02:31:48.069605 26244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:31:48.977873 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:35915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40897
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:31:49 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 02:31:49.723330 26279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:49.723940 26279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:49.724232 26279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:49.741474 26279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:49.741781 26279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:49.742094 26279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:31:49.763845 26279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.8.212:35915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:31:49.766129 26279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:49.768431 26279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:49.805471 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:49.809931 26293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:49.815397 26295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:49.821003 26279 server_base.cc:1034] running on GCE node
I20260812 02:31:49.822028 26279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:49.823509 26279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:49.824934 26279 hybrid_clock.cc:648] HybridClock initialized: now 1786501909824704 us; error 101 us; skew 500 ppm
I20260812 02:31:49.825736 26279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:31:49 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:49.831116 26279 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:31:49.846194 26279 webserver.cc:460] Webserver started at http://127.12.8.193:42301/ using document root <none> and password file <none>
I20260812 02:31:49.847610 26279 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:49.847865 26279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:49.848474 26279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:31:49 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:31:49.879462 26279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "3e946a87db6c422ea32d1c78ba8d42ba"
format_stamp: "Formatted at 2026-08-12 02:31:49 on dist-test-slave-vq2q"
server_key: "47f5bdcec154c77284922803c0953a18"
server_key_iv: "66d860386c5fcbd84f0734536b92369c"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:49.881315 26279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "3e946a87db6c422ea32d1c78ba8d42ba"
format_stamp: "Formatted at 2026-08-12 02:31:49 on dist-test-slave-vq2q"
server_key: "47f5bdcec154c77284922803c0953a18"
server_key_iv: "66d860386c5fcbd84f0734536b92369c"
server_key_version: "kuduclusterkey@0"
Aug 12 02:31:49 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:31:49.918620 26279 fs_manager.cc:691] Time spent creating directory manager: real 0.037s user 0.021s sys 0.012s
I20260812 02:31:49.961548 26303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:49.963842 26279 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.002s sys 0.014s
I20260812 02:31:49.964119 26279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "3e946a87db6c422ea32d1c78ba8d42ba"
format_stamp: "Formatted at 2026-08-12 02:31:49 on dist-test-slave-vq2q"
server_key: "47f5bdcec154c77284922803c0953a18"
server_key_iv: "66d860386c5fcbd84f0734536b92369c"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:49.964499 26279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:50.030390 26279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:50.036819 26279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:50.037397 26279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:50.039312 26279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:50.042193 26279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:50.042371 26279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:50.042603 26279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:50.042732 26279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:50.154263 26279 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:43725
I20260812 02:31:50.154323 26439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:43725 every 8 connection(s)
I20260812 02:31:50.155791 26279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:31:50.164561 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 26279
I20260812 02:31:50.164840 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:31:50.165796 12323 external_mini_cluster.cc:1428] Setting key 6ddf97e4eb7eed58aeb80229eabf1032
Aug 12 02:31:50 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:31:50.214403 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.160139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40897 (local address 127.12.8.193:58911)
0812 02:31:50.182759 (+ 22620us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.182848 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.187432 (+ 4584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.191715 (+ 4283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.191727 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.192511 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:50.193514 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.193538 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.194856 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.194860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.196811 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.196825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.198167 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.198176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.198500 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.199404 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:50.199430 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:50.202582 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:50.208592 (+ 6010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.208601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.208616 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.209246 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.209826 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.209831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.209834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.210208 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.211038 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:50.211047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:50.211537 (+ 490us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.211850 (+ 313us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.212573 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22126,"thread_start_us":28390,"threads_started":1}
I20260812 02:31:50.216698 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.161030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:58911 (local address 127.12.8.254:40897)
0812 02:31:50.181668 (+ 20638us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.181677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.188120 (+ 6443us) server_negotiation.cc:407] Connection header received
0812 02:31:50.188220 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.188227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.188355 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.188472 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:50.193731 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.194666 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.196966 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.197782 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.197921 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.202951 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:50.202958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:50.202962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:50.203008 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:50.208405 (+ 5397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.209419 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.209425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.209428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.209689 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.210370 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.210374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.210376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.210882 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:50.215675 (+ 4793us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.215850 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.216151 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20382,"thread_start_us":20659,"threads_started":1}
I20260812 02:31:50.218858 26440 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40897
I20260812 02:31:50.219292 26440 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:50.220574 26440 heartbeater.cc:511] Master 127.12.8.254:40897 requested a full tablet report, sending...
I20260812 02:31:50.223888 26184 ts_manager.cc:194] Registered new tserver with Master: 3e946a87db6c422ea32d1c78ba8d42ba (127.12.8.193:43725)
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
I20260812 02:31:50.226624 26184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:58911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:31:50.269783 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:35915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40897
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:31:50 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
W20260812 02:31:50.971309 26448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:50.971632 26448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:50.971870 26448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:50.981230 26448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:50.981405 26448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:50.981590 26448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:31:50.992743 26448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.8.212:35915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:31:50.994225 26448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:50.995817 26448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:51.013360 26462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:51.013381 26463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:51.014443 26465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:51.015511 26448 server_base.cc:1034] running on GCE node
I20260812 02:31:51.016173 26448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:51.017407 26448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:51.018658 26448 hybrid_clock.cc:648] HybridClock initialized: now 1786501911018585 us; error 51 us; skew 500 ppm
I20260812 02:31:51.019004 26448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:31:51 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:51.022863 26448 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:31:51.024564 26448 webserver.cc:460] Webserver started at http://127.12.8.194:34903/ using document root <none> and password file <none>
I20260812 02:31:51.025435 26448 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:51.025640 26448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:51.025995 26448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:31:51 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:31:51.045791 26448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "09afdf6263a54ce5b4eb311bc43d4d86"
format_stamp: "Formatted at 2026-08-12 02:31:51 on dist-test-slave-vq2q"
server_key: "191206fb29cbe5c415fa1dbb6287f9ef"
server_key_iv: "439831359383251dfbf65231941037dd"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:51.047513 26448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "09afdf6263a54ce5b4eb311bc43d4d86"
format_stamp: "Formatted at 2026-08-12 02:31:51 on dist-test-slave-vq2q"
server_key: "191206fb29cbe5c415fa1dbb6287f9ef"
server_key_iv: "439831359383251dfbf65231941037dd"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:51.056787 26448 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 02:31:51.077718 26474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:51.079481 26448 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 02:31:51.079710 26448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "09afdf6263a54ce5b4eb311bc43d4d86"
format_stamp: "Formatted at 2026-08-12 02:31:51 on dist-test-slave-vq2q"
server_key: "191206fb29cbe5c415fa1dbb6287f9ef"
server_key_iv: "439831359383251dfbf65231941037dd"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:51.080035 26448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:51.096724 26448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:51.102082 26448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:51.102439 26448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:51.103734 26448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:51.106057 26448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:51.106163 26448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:51.106294 26448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:51.106379 26448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:51.152441 26448 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:44809
I20260812 02:31:51.152519 26614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:44809 every 8 connection(s)
I20260812 02:31:51.153920 26448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:31:51.157570 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 26448
I20260812 02:31:51.157783 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:31:51.158475 12323 external_mini_cluster.cc:1428] Setting key 33382cd103e1cfee3fd0379148add3c5
Aug 12 02:31:51 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:31:51.181097 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.157070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:45609 (local address 127.12.8.254:40897)
0812 02:31:51.160713 (+ 3643us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.160719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.162288 (+ 1569us) server_negotiation.cc:407] Connection header received
0812 02:31:51.164498 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.164504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.164587 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.164699 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:51.166223 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.167379 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.169427 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.170126 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.170254 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.174226 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:51.174232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:51.174236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:51.174276 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:51.177344 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:51.178299 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:51.178304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:51.178306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:51.178451 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:51.179043 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:51.179047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:51.179049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:51.179349 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:51.179485 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.180439 (+ 954us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.180643 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3495,"thread_start_us":3377,"threads_started":1}
I20260812 02:31:51.182550 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.157577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40897 (local address 127.12.8.194:45609)
0812 02:31:51.162108 (+ 4531us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.162161 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.164199 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.164906 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.164914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.165426 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:51.166056 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.166075 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.167519 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.167523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.169291 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.169303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.170409 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.170416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.170770 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.171487 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:51.171509 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:51.173992 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:51.177513 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:51.177522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:51.177539 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:51.178128 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:51.178562 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:51.178567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:51.178570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:51.178908 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:51.179473 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:51.179480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:51.179958 (+ 478us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.180306 (+ 348us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.180971 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4223,"thread_start_us":3930,"threads_started":1}
I20260812 02:31:51.184410 26615 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40897
I20260812 02:31:51.184732 26615 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:51.185738 26615 heartbeater.cc:511] Master 127.12.8.254:40897 requested a full tablet report, sending...
I20260812 02:31:51.187449 26184 ts_manager.cc:194] Registered new tserver with Master: 09afdf6263a54ce5b4eb311bc43d4d86 (127.12.8.194:44809)
I20260812 02:31:51.188607 26184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:45609
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
I20260812 02:31:51.230809 26440 heartbeater.cc:503] Master 127.12.8.254:40897 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:31:51.232899 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:35915/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40897
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:51.795076 26622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:51.795509 26622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:51.795755 26622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:51.807013 26622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:51.807250 26622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:51.807523 26622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:31:51.822494 26622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.8.212:35915/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:31:51.824167 26622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:51.826048 26622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:51.847427 26627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:51.847587 26628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:51.848299 26630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:51.850054 26622 server_base.cc:1034] running on GCE node
I20260812 02:31:51.851127 26622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:51.852584 26622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:51.854018 26622 hybrid_clock.cc:648] HybridClock initialized: now 1786501911853859 us; error 83 us; skew 500 ppm
I20260812 02:31:51.854584 26622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:31:51 dist-test-slave-vq2q krb5kdc[25603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:51.871299 26622 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:31:51.873662 26622 webserver.cc:460] Webserver started at http://127.12.8.195:38575/ using document root <none> and password file <none>
I20260812 02:31:51.874558 26622 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:51.874680 26622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:51.875003 26622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:31:51 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:31:51.896646 26622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "1c264096afbe4ba3a49fca8c719f9ec9"
format_stamp: "Formatted at 2026-08-12 02:31:51 on dist-test-slave-vq2q"
server_key: "9152e9859fae6cbd4d0157550d159aaf"
server_key_iv: "8139334406e002cfcf4451b7e6e5d60f"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:51.897938 26622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "1c264096afbe4ba3a49fca8c719f9ec9"
format_stamp: "Formatted at 2026-08-12 02:31:51 on dist-test-slave-vq2q"
server_key: "9152e9859fae6cbd4d0157550d159aaf"
server_key_iv: "8139334406e002cfcf4451b7e6e5d60f"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:51.906368 26622 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s
I20260812 02:31:51.927448 26638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:51.929359 26622 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260812 02:31:51.929697 26622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "1c264096afbe4ba3a49fca8c719f9ec9"
format_stamp: "Formatted at 2026-08-12 02:31:51 on dist-test-slave-vq2q"
server_key: "9152e9859fae6cbd4d0157550d159aaf"
server_key_iv: "8139334406e002cfcf4451b7e6e5d60f"
server_key_version: "kuduclusterkey@0"
I20260812 02:31:51.930058 26622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:51.969987 26622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:51.977255 26622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:51.977917 26622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:51.980109 26622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:51.983204 26622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:51.983366 26622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:51.983634 26622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:51.983752 26622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:52.053857 26622 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:46119
I20260812 02:31:52.053968 26771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:46119 every 8 connection(s)
I20260812 02:31:52.055672 26622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:31:52.064796 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 26622
I20260812 02:31:52.065012 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:31:52.065611 12323 external_mini_cluster.cc:1428] Setting key bb78c3afb5844697672b7d7f273fb085
Aug 12 02:31:52 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:31:52.083545 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.059624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:37967 (local address 127.12.8.254:40897)
0812 02:31:52.060381 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.060385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.062011 (+ 1626us) server_negotiation.cc:407] Connection header received
0812 02:31:52.064390 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.064396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.064499 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.064607 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:52.066770 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.068241 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.070545 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.071087 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.071262 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.075659 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:52.075666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:52.075670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:52.075747 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:52.079154 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.080035 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.080081 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.080085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.080317 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.081127 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.081133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.081137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.081723 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:52.081909 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.082772 (+ 863us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.083103 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":514,"threads_started":1}
I20260812 02:31:52.084604 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.060470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40897 (local address 127.12.8.195:37967)
0812 02:31:52.061776 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:52.061872 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:52.064068 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:52.064915 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:52.064927 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:52.065665 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:52.066529 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.066559 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.068424 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.068428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.070339 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.070361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.071394 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.071402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.071683 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.072578 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:52.072614 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:52.075299 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:52.079349 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:52.079356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:52.079370 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:52.079848 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:52.080479 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:52.080483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:52.080485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:52.080965 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:52.081900 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:52.081907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:52.082368 (+ 461us) client_negotiation.cc:769] Sending connection context
0812 02:31:52.082672 (+ 304us) client_negotiation.cc:240] Negotiation successful
0812 02:31:52.083326 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":623,"threads_started":1}
I20260812 02:31:52.086623 26772 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40897
I20260812 02:31:52.086969 26772 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:52.087769 26772 heartbeater.cc:511] Master 127.12.8.254:40897 requested a full tablet report, sending...
I20260812 02:31:52.089686 26184 ts_manager.cc:194] Registered new tserver with Master: 1c264096afbe4ba3a49fca8c719f9ec9 (127.12.8.195:46119)
I20260812 02:31:52.090626 26184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:37967
I20260812 02:31:52.096760 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:31:52.121352 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.104356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.12.8.254:40897)
0812 02:31:52.104561 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.104566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.105294 (+ 728us) server_negotiation.cc:407] Connection header received
0812 02:31:52.105488 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.105492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.105586 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.105713 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:52.107658 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.108640 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.110518 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.111171 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.111295 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.114725 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:52.114730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:52.114734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:52.114770 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:52.118050 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.119032 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.119036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.119039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.119176 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.119709 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.119712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.119714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.119997 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:52.120235 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.120749 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.120975 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:31:52.127238 26184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:31:52.135691 26184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:52.183418 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.156915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.12.8.195:46119)
0812 02:31:52.171539 (+ 14624us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.171549 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.171591 (+ 42us) server_negotiation.cc:407] Connection header received
0812 02:31:52.171715 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.171724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.172043 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.172265 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:52.173804 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.175876 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.179350 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.181720 (+ 2370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.181876 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.182247 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.182676 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.182926 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14442,"thread_start_us":14224,"threads_started":1}
I20260812 02:31:52.183447 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.156373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:46119 (local address 127.0.0.1:37934)
0812 02:31:52.170331 (+ 13958us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:52.170357 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:52.170550 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:52.172551 (+ 2001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:52.172561 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:52.172596 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:52.173365 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.173380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.176094 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.176100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.179119 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.179138 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.182036 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.182045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.182291 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.182357 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 02:31:52.182527 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 02:31:52.182741 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13775,"mutex_wait_us":1,"thread_start_us":19722,"threads_started":1}
I20260812 02:31:52.186896 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.152824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.12.8.194:44809)
0812 02:31:52.176225 (+ 23401us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.176236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.176280 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:31:52.176412 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.176421 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.176771 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.177073 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:52.178515 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.179951 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.183213 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.184524 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.185303 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.185890 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.186049 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.186340 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23165,"thread_start_us":22960,"threads_started":1}
I20260812 02:31:52.187508 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.152590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:44809 (local address 127.0.0.1:56240)
0812 02:31:52.153822 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:52.153868 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:52.154089 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:52.177073 (+ 22984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:52.177080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:52.177115 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:52.178086 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.178102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.180132 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.180139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.183000 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.183023 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.184836 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.184845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.185098 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.185140 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 02:31:52.185328 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 02:31:52.185479 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":61,"thread_start_us":692,"threads_started":1}
I20260812 02:31:52.188954 26693 tablet_service.cc:1467] Processing CreateTablet for tablet 14725a40dae44ac3920dca7473f93e79 (DEFAULT_TABLE table=test-table [id=4e16c786cfa5448a8ec0d4ec311b4d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:52.190769 26693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14725a40dae44ac3920dca7473f93e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:52.191604 26615 heartbeater.cc:503] Master 127.12.8.254:40897 was elected leader, sending a full tablet report...
I20260812 02:31:52.192377 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.158697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:43725 (local address 127.0.0.1:41150)
0812 02:31:52.176140 (+ 17443us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:52.176160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:52.176303 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:52.177683 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:52.177694 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:52.177722 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:52.178562 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.178574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.180984 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.180990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.185944 (+ 4954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.185982 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.189265 (+ 3283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.189274 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.190895 (+ 1621us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.190955 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 02:31:52.191068 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 02:31:52.191225 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17308,"mutex_wait_us":25,"thread_start_us":11944,"threads_started":1}
I20260812 02:31:52.192926 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.158855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.12.8.193:43725)
0812 02:31:52.176983 (+ 18128us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.176992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.177023 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:31:52.177130 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.177139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.177424 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.177626 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:52.178769 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.180764 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.186289 (+ 5525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.188507 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.188681 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.188965 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.191172 (+ 2207us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.191477 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17922,"thread_start_us":17776,"threads_started":1}
I20260812 02:31:52.193760 26538 tablet_service.cc:1467] Processing CreateTablet for tablet 14725a40dae44ac3920dca7473f93e79 (DEFAULT_TABLE table=test-table [id=4e16c786cfa5448a8ec0d4ec311b4d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:52.196234 26538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14725a40dae44ac3920dca7473f93e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:52.198309 26360 tablet_service.cc:1467] Processing CreateTablet for tablet 14725a40dae44ac3920dca7473f93e79 (DEFAULT_TABLE table=test-table [id=4e16c786cfa5448a8ec0d4ec311b4d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:52.200515 26360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14725a40dae44ac3920dca7473f93e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:52.206074 26798 tablet_bootstrap.cc:492] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Bootstrap starting.
I20260812 02:31:52.210400 26798 tablet_bootstrap.cc:654] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:52.211565 26798 log.cc:824] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:52.215583 26798 tablet_bootstrap.cc:492] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: No bootstrap required, opened a new log
I20260812 02:31:52.215896 26798 ts_tablet_manager.cc:1397] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260812 02:31:52.216704 26800 tablet_bootstrap.cc:492] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Bootstrap starting.
I20260812 02:31:52.220873 26800 tablet_bootstrap.cc:654] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:52.222894 26800 log.cc:824] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:52.223461 26803 tablet_bootstrap.cc:492] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Bootstrap starting.
I20260812 02:31:52.226689 26798 raft_consensus.cc:348] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.227197 26798 raft_consensus.cc:374] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:52.227315 26800 tablet_bootstrap.cc:492] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: No bootstrap required, opened a new log
I20260812 02:31:52.227363 26798 raft_consensus.cc:729] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c264096afbe4ba3a49fca8c719f9ec9, State: Initialized, Role: FOLLOWER
I20260812 02:31:52.227638 26800 ts_tablet_manager.cc:1397] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260812 02:31:52.227880 26803 tablet_bootstrap.cc:654] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:52.228078 26798 consensus_queue.cc:260] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.229427 26803 log.cc:824] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:52.229557 26772 heartbeater.cc:503] Master 127.12.8.254:40897 was elected leader, sending a full tablet report...
I20260812 02:31:52.230221 26798 ts_tablet_manager.cc:1428] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260812 02:31:52.233695 26803 tablet_bootstrap.cc:492] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: No bootstrap required, opened a new log
I20260812 02:31:52.234146 26803 ts_tablet_manager.cc:1397] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260812 02:31:52.238225 26800 raft_consensus.cc:348] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.238780 26800 raft_consensus.cc:374] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:52.238878 26800 raft_consensus.cc:729] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09afdf6263a54ce5b4eb311bc43d4d86, State: Initialized, Role: FOLLOWER
I20260812 02:31:52.239605 26800 consensus_queue.cc:260] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.241387 26800 ts_tablet_manager.cc:1428] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260812 02:31:52.245934 26803 raft_consensus.cc:348] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.246601 26803 raft_consensus.cc:374] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:52.246738 26803 raft_consensus.cc:729] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e946a87db6c422ea32d1c78ba8d42ba, State: Initialized, Role: FOLLOWER
I20260812 02:31:52.247572 26803 consensus_queue.cc:260] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.249912 26803 ts_tablet_manager.cc:1428] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
W20260812 02:31:52.308595 26773 tablet.cc:2369] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:31:52.406797 26617 tablet.cc:2369] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:31:52.438906 26809 raft_consensus.cc:482] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:52.439303 26809 raft_consensus.cc:504] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.441500 26809 leader_election.cc:288] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09afdf6263a54ce5b4eb311bc43d4d86 (127.12.8.194:44809), 1c264096afbe4ba3a49fca8c719f9ec9 (127.12.8.195:46119)
W20260812 02:31:52.443581 26441 tablet.cc:2369] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:31:52.450053 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.442672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:54873 (local address 127.12.8.195:46119)
0812 02:31:52.442954 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.442960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.443747 (+ 787us) server_negotiation.cc:407] Connection header received
0812 02:31:52.444022 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.444028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.444124 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.444214 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:52.445100 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.445939 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.448161 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.449031 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.449243 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.449342 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.449657 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.449793 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260812 02:31:52.450066 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.442047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:40315 (local address 127.12.8.194:44809)
0812 02:31:52.442426 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.442435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.443474 (+ 1039us) server_negotiation.cc:407] Connection header received
0812 02:31:52.443658 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.443663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.443753 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.443849 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:52.444836 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.445771 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.448030 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.448953 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.449119 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.449256 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.449602 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.449759 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260812 02:31:52.450191 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.441979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:44809 (local address 127.12.8.193:40315)
0812 02:31:52.443107 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:52.443139 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:52.443469 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:52.444032 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:52.444037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:52.444074 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:52.444650 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.444664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.445933 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.445938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.447837 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.447851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.449221 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.449228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.449415 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.449446 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:31:52.449533 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 02:31:52.449668 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":86,"thread_start_us":616,"threads_started":1}
I20260812 02:31:52.450284 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.442289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:46119 (local address 127.12.8.193:54873)
0812 02:31:52.443580 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:52.443603 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:52.443821 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:52.444292 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:52.444298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:52.444333 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:52.444878 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.444892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.446114 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.446119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.447982 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:52.447999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.449335 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.449343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:52.449476 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.449502 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:31:52.449624 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 02:31:52.449775 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":1,"thread_start_us":784,"threads_started":1}
I20260812 02:31:52.451413 26562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14725a40dae44ac3920dca7473f93e79" candidate_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" is_pre_election: true
I20260812 02:31:52.451486 26720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14725a40dae44ac3920dca7473f93e79" candidate_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" is_pre_election: true
I20260812 02:31:52.452056 26562 raft_consensus.cc:2454] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e946a87db6c422ea32d1c78ba8d42ba in term 0.
I20260812 02:31:52.452122 26720 raft_consensus.cc:2454] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e946a87db6c422ea32d1c78ba8d42ba in term 0.
I20260812 02:31:52.452860 26305 leader_election.cc:302] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09afdf6263a54ce5b4eb311bc43d4d86, 3e946a87db6c422ea32d1c78ba8d42ba; no voters:
I20260812 02:31:52.453380 26809 raft_consensus.cc:2781] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:52.453557 26809 raft_consensus.cc:482] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:52.453667 26809 raft_consensus.cc:3037] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:52.456332 26809 raft_consensus.cc:504] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.456964 26809 leader_election.cc:288] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [CANDIDATE]: Term 1 election: Requested vote from peers 09afdf6263a54ce5b4eb311bc43d4d86 (127.12.8.194:44809), 1c264096afbe4ba3a49fca8c719f9ec9 (127.12.8.195:46119)
I20260812 02:31:52.457612 26720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14725a40dae44ac3920dca7473f93e79" candidate_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c264096afbe4ba3a49fca8c719f9ec9"
I20260812 02:31:52.457617 26562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14725a40dae44ac3920dca7473f93e79" candidate_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09afdf6263a54ce5b4eb311bc43d4d86"
I20260812 02:31:52.457839 26720 raft_consensus.cc:3037] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:52.457902 26562 raft_consensus.cc:3037] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:52.460353 26720 raft_consensus.cc:2454] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e946a87db6c422ea32d1c78ba8d42ba in term 1.
I20260812 02:31:52.460547 26562 raft_consensus.cc:2454] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e946a87db6c422ea32d1c78ba8d42ba in term 1.
I20260812 02:31:52.460942 26306 leader_election.cc:302] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c264096afbe4ba3a49fca8c719f9ec9, 3e946a87db6c422ea32d1c78ba8d42ba; no voters:
I20260812 02:31:52.461365 26809 raft_consensus.cc:2781] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:52.462025 26809 raft_consensus.cc:686] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 1 LEADER]: Becoming Leader. State: Replica: 3e946a87db6c422ea32d1c78ba8d42ba, State: Running, Role: LEADER
I20260812 02:31:52.462648 26809 consensus_queue.cc:237] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } }
I20260812 02:31:52.468843 26184 catalog_manager.cc:5495] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e946a87db6c422ea32d1c78ba8d42ba (127.12.8.193). New cstate: current_term: 1 leader_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e946a87db6c422ea32d1c78ba8d42ba" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20260812 02:31:52.543195 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.536879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.12.8.193:43725)
0812 02:31:52.537153 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.537159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.537212 (+ 53us) server_negotiation.cc:407] Connection header received
0812 02:31:52.537339 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.537343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.537435 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.537586 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:52.537982 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.538735 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.540471 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.541045 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.541153 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.541931 (+ 778us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:52.542503 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:52.542619 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.542761 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.542879 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 02:31:52.553812 26562 raft_consensus.cc:1261] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 1 FOLLOWER]: Refusing update from remote peer 3e946a87db6c422ea32d1c78ba8d42ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:52.553812 26720 raft_consensus.cc:1261] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 1 FOLLOWER]: Refusing update from remote peer 3e946a87db6c422ea32d1c78ba8d42ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:52.555235 26812 consensus_queue.cc:1035] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c264096afbe4ba3a49fca8c719f9ec9" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:52.555670 26809 consensus_queue.cc:1035] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "09afdf6263a54ce5b4eb311bc43d4d86" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:52.587915 26183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:52.588207 26183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:52.592399 26183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf6fbdce6ec342e1b7d4a96cf7592ea8: Sending DeleteTablet for 3 replicas of tablet 14725a40dae44ac3920dca7473f93e79
I20260812 02:31:52.593815 26360 tablet_service.cc:1514] Processing DeleteTablet for tablet 14725a40dae44ac3920dca7473f93e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:52 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:41150
I20260812 02:31:52.594000 26538 tablet_service.cc:1514] Processing DeleteTablet for tablet 14725a40dae44ac3920dca7473f93e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:52 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:56240
I20260812 02:31:52.594310 26693 tablet_service.cc:1514] Processing DeleteTablet for tablet 14725a40dae44ac3920dca7473f93e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:52 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:37934
I20260812 02:31:52.597774 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 26279
I20260812 02:31:52.607677 26829 tablet_replica.cc:330] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: stopping tablet replica
I20260812 02:31:52.608501 26829 raft_consensus.cc:2229] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:31:52.609649 26829 raft_consensus.cc:2258] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:52.609921 26830 tablet_replica.cc:330] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: stopping tablet replica
I20260812 02:31:52.610190 26833 tablet_replica.cc:330] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: stopping tablet replica
I20260812 02:31:52.610653 26830 raft_consensus.cc:2229] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:52.610780 26833 raft_consensus.cc:2229] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:52.611696 26833 raft_consensus.cc:2258] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:52.611683 26830 raft_consensus.cc:2258] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:52.613103 26829 ts_tablet_manager.cc:1905] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:52.614758 26833 ts_tablet_manager.cc:1905] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:52.614758 26830 ts_tablet_manager.cc:1905] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:52.620959 26829 ts_tablet_manager.cc:1918] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:31:52.621294 26829 log.cc:1196] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/14725a40dae44ac3920dca7473f93e79
I20260812 02:31:52.621994 26829 ts_tablet_manager.cc:1939] T 14725a40dae44ac3920dca7473f93e79 P 3e946a87db6c422ea32d1c78ba8d42ba: Deleting consensus metadata
I20260812 02:31:52.622942 26830 ts_tablet_manager.cc:1918] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:31:52.622938 26833 ts_tablet_manager.cc:1918] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:31:52.623186 26833 log.cc:1196] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/14725a40dae44ac3920dca7473f93e79
I20260812 02:31:52.623857 26833 ts_tablet_manager.cc:1939] T 14725a40dae44ac3920dca7473f93e79 P 1c264096afbe4ba3a49fca8c719f9ec9: Deleting consensus metadata
I20260812 02:31:52.624286 26163 catalog_manager.cc:4841] TS 3e946a87db6c422ea32d1c78ba8d42ba (127.12.8.193:43725): tablet 14725a40dae44ac3920dca7473f93e79 (table test-table [id=4e16c786cfa5448a8ec0d4ec311b4d42]) successfully deleted
I20260812 02:31:52.624850 26830 log.cc:1196] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/14725a40dae44ac3920dca7473f93e79
I20260812 02:31:52.625651 26830 ts_tablet_manager.cc:1939] T 14725a40dae44ac3920dca7473f93e79 P 09afdf6263a54ce5b4eb311bc43d4d86: Deleting consensus metadata
I20260812 02:31:52.626742 26165 catalog_manager.cc:4841] TS 1c264096afbe4ba3a49fca8c719f9ec9 (127.12.8.195:46119): tablet 14725a40dae44ac3920dca7473f93e79 (table test-table [id=4e16c786cfa5448a8ec0d4ec311b4d42]) successfully deleted
I20260812 02:31:52.627948 26164 catalog_manager.cc:4841] TS 09afdf6263a54ce5b4eb311bc43d4d86 (127.12.8.194:44809): tablet 14725a40dae44ac3920dca7473f93e79 (table test-table [id=4e16c786cfa5448a8ec0d4ec311b4d42]) successfully deleted
Aug 12 02:31:52 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:31:52.636039 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.599622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.12.8.193:43725)
0812 02:31:52.599862 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.599870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.615603 (+ 15733us) server_negotiation.cc:407] Connection header received
0812 02:31:52.616139 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.616145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.616246 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.616322 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:52.618358 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.619563 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.621501 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.622158 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.622266 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.629476 (+ 7210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:52.629493 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:52.629509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:52.629566 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:52.632801 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.633636 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.633644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.633646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.633821 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.634216 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.634219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.634221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.634614 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:52.634965 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.635263 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.635507 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:31:52.636786 26434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:31:52.752653 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 26279
I20260812 02:31:52.776408 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 26448
Aug 12 02:31:52 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:31:52.793609 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.777824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.12.8.194:44809)
0812 02:31:52.778092 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.778099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.778124 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:31:52.778284 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.778289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.778397 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.778492 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:52.779886 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.780652 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.782031 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.782454 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.782567 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.787480 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:52.787499 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:52.787515 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:52.787555 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:52.790590 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.791292 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.791300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.791302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.791440 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.791838 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.791842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.791844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.792245 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:52.792532 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.792841 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.793120 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 02:31:52.794297 26610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:31:52.890398 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 26448
I20260812 02:31:52.918839 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 26622
Aug 12 02:31:52 dist-test-slave-vq2q krb5kdc[25603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:31:52.936781 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:52.919799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.12.8.195:46119)
0812 02:31:52.920069 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:52.920075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:52.920099 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:31:52.920226 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:52.920231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:52.920335 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:52.920435 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:52.921886 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.922783 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.924043 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:52.924501 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:52.924613 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:52.929551 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:52.929589 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:52.929611 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:52.929672 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:52.933136 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.934002 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.934012 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.934014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.934347 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:52.934915 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:52.934922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:52.934924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:52.935517 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:52.935855 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:52.936138 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 02:31:52.936429 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260812 02:31:52.937664 26765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:31:53.042133 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 26622
I20260812 02:31:53.066599 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 26146
I20260812 02:31:53.068305 26224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:31:53.188163 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 26146
I20260812 02:31:53.219183 12323 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 02:31:53.566833 12323 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 02:31:53.567157 12323 mini_ranger.cc:66] Stopping Ranger...
I20260812 02:31:53.925354 12323 mini_ranger.cc:68] Stopped Ranger
2026-08-12 02:31:53.925 UTC [25643] LOG: received smart shutdown request
2026-08-12 02:31:53.930 UTC [25643] LOG: background worker "logical replication launcher" (PID 25654) exited with exit code 1
2026-08-12 02:31:53.930 UTC [25647] LOG: shutting down
2026-08-12 02:31:54.096 UTC [25643] LOG: database system is shut down
2026-08-12T02:31:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57786 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:31:54 dist-test-slave-vq2q krb5kdc[26851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:31:54 dist-test-slave-vq2q krb5kdc[26851](info): set up 2 sockets
Aug 12 02:31:54 dist-test-slave-vq2q krb5kdc[26851](info): commencing operation
krb5kdc: starting...
W20260812 02:31:55.421161 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.002s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:31:55 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:31:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.8.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.8.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/rangeradmin_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/rangeradmin_127.12.8.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.8.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.8.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/rangerlookup_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/rangerlookup_127.12.8.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.8.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.8.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/HTTP_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/HTTP_127.12.8.212@KRBTEST.COM.keytab.
I20260812 02:31:55.922613 12323 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqIRTh5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/postgres -l logfile start
2026-08-12 02:31:58.653 UTC [26891] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 02:31:58.653 UTC [26891] LOG: listening on IPv4 address "127.12.8.212", port 37087
2026-08-12 02:31:58.658 UTC [26891] LOG: listening on Unix socket "/tmp/.s.PGSQL.37087"
2026-08-12 02:31:58.681 UTC [26893] LOG: database system was shut down at 2026-08-12 02:31:57 UTC
2026-08-12 02:31:58.685 UTC [26891] LOG: database system is ready to accept connections
I20260812 02:31:59.675596 12323 mini_postgres.cc:96] Postgres bound to 37087
2026-08-12 02:31:59.700 UTC [26903] FATAL: database "slave" does not exist
127.12.8.212:37087 - accepting connections
I20260812 02:31:59.700929 12323 mini_ranger.cc:161] Starting Ranger...
I20260812 02:31:59.734970 12323 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 02:32:00.281418 12323 mini_ranger.cc:87] Created ranger Postgres database
I20260812 02:32:00.281536 12323 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-admin
2026-08-12 02:32:00,583 [I] DB FLAVOR :POSTGRES
2026-08-12 02:32:00,583 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:32:00,584 [I] Checking connection..
2026-08-12 02:32:00,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 02:32:00,852 [I] Checking connection passed.
2026-08-12 02:32:00,852 [I] --------- Verifying version history table ---------
2026-08-12 02:32:00,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:32:01,115 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:32:01,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:32:01,384 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:32:01,384 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 02:32:01,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 02:32:01.646 UTC [26991] WARNING: there is no transaction in progress
2026-08-12 02:32:01,653 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 02:32:01,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:32:01,920 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 02:32:01,921 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:32:01,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 02:32:02,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 02:32:02,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vq2q.c.gcp-upstream.internal','N') ;"
2026-08-12 02:32:02,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:32:02,942 [I] Table x_portal_user does not exist in database ranger
2026-08-12 02:32:02,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:32:03,191 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 02:32:03,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:32:03,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:32:03,707 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 02:32:03,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 02:32:04.724 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.742 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.761 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.778 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.796 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.819 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.879 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.887 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.903 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.915 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.926 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.941 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.952 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.958 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.966 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:04.973 UTC [27127] WARNING: there is no transaction in progress
2026-08-12 02:32:05,660 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 02:32:05,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 02:32:05,957 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 02:32:05,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:32:06,270 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 02:32:06,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:32:06,555 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 02:32:06,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:32:06,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:32:07,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vq2q.c.gcp-upstream.internal';"
2026-08-12 02:32:07,364 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 02:32:07,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 02:32:07,650 [I] DB_PATCHES have already been applied
I20260812 02:32:07.656493 12323 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqIRTh5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 02:32:07.656632 12323 mini_ranger.cc:193] Using host: 127.12.8.212
I20260812 02:32:07.685853 12323 mini_ranger.cc:227] Ranger admin URL: http://127.12.8.212:45507
Aug 12, 2026 2:32:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:32:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:32:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:32:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:32:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:32:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:32:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 02:32:08.926465 12323 mini_ranger.cc:160] Time spent starting Ranger: real 9.226s user 0.002s sys 0.085s
Aug 12, 2026 2:32:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:32:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:32:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:32:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:32:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 02:32:33.660291 12323 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.8.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.8.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/rangerkms_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.8.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/rangerkms_127.12.8.212@KRBTEST.COM.keytab.
W20260812 02:32:33.763732 12323 mini_kdc.cc:262] Time spent creating service keytab for rangerkms/127.12.8.212@KRBTEST.COM: real 0.102s user 0.002s sys 0.032s
WARNING: no policy specified for HTTP/127.12.8.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.8.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.8.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/HTTP_127.12.8.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.8.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/HTTP_127.12.8.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 02:32:33 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501953, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 02:32:33.944756 12323 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 02:32:33.975109 12323 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 02:32:34.956156 12323 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 02:32:34.956313 12323 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms
I20260812 02:32:34.956355 12323 mini_ranger_kms.cc:227] Using postgres at 127.12.8.212:37087
2026-08-12 02:32:35,385 [I] DB FLAVOR :POSTGRES
2026-08-12 02:32:35,386 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:32:35,386 [I] Checking connection
2026-08-12 02:32:35,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 02:32:35,642 [I] connection success
2026-08-12 02:32:35,643 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 02:32:35,643 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 02:32:35,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 02:32:35,932 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 02:32:35,933 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:32:35,933 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 02:32:35,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.212:37087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 02:32:36,264 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 02:32:36.891222 12323 mini_ranger_kms.cc:310] Created kms service
I20260812 02:32:37.106721 12323 mini_ranger_kms.cc:326] Created kudu user
I20260812 02:32:37.164412 12323 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 02:32:37 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501953, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:37.673424 12323 mini_ranger_kms.cc:384] Added ranger policy
I20260812 02:32:37.673784 12323 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqIRTh5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqIRTh5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 02:32:37.673822 12323 mini_ranger_kms.cc:240] Using host: 127.12.8.212
I20260812 02:32:37.702850 12323 mini_ranger_kms.cc:276] Ranger KMS PID: 27353
I20260812 02:32:37.703024 12323 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.8.212:53507
Aug 12, 2026 2:32:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:32:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:32:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:32:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:32:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:32:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 02:32:40.042038 12323 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.097s user 0.069s sys 0.415s
I20260812 02:32:40.042361 12323 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 02:32:40.042587 12323 mini_ranger_kms.cc:401] 127.12.8.212:53507/kms/v1/keys
Aug 12 02:32:40 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786501960, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 02:32:40 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501960, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.212@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
Aug 12 02:32:41 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501960, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.212@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
Aug 12 02:32:43 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:32:44.034756 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40263
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40263
--ranger_config_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:53507/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:44.404443 27411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:44.404843 27411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:44.404999 27411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:44.412844 27411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:44.412971 27411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:44.413049 27411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:44.413164 27411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:44.413290 27411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:44.423985 27411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.8.212:53507/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40263
--ranger_config_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:44.425444 27411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:44.426853 27411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:44.439839 27416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:44.440912 27418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:44.445535 27420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:44.446131 27411 server_base.cc:1034] running on GCE node
I20260812 02:32:44.447039 27411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:44.448680 27411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:44.449957 27411 hybrid_clock.cc:648] HybridClock initialized: now 1786501964449899 us; error 58 us; skew 500 ppm
I20260812 02:32:44.450337 27411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:44 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501964, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:44.457732 27411 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:32:44.459604 27411 webserver.cc:460] Webserver started at http://127.12.8.254:37693/ using document root <none> and password file <none>
I20260812 02:32:44.460400 27411 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:44.460593 27411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:44.460958 27411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:32:44 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501964, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:44.544205 27411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "35af554318784f17a72635501ca5e8fb"
format_stamp: "Formatted at 2026-08-12 02:32:44 on dist-test-slave-vq2q"
server_key: "90fd7573f0e16b04c5608c253dac8c61"
server_key_iv: "b9ab24d5699834a9b066926293699284"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:44.545121 27411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "35af554318784f17a72635501ca5e8fb"
format_stamp: "Formatted at 2026-08-12 02:32:44 on dist-test-slave-vq2q"
server_key: "90fd7573f0e16b04c5608c253dac8c61"
server_key_iv: "b9ab24d5699834a9b066926293699284"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:44.551290 27411 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:32:44.574265 27433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:44.576700 27411 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:32:44.577009 27411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "35af554318784f17a72635501ca5e8fb"
format_stamp: "Formatted at 2026-08-12 02:32:44 on dist-test-slave-vq2q"
server_key: "90fd7573f0e16b04c5608c253dac8c61"
server_key_iv: "b9ab24d5699834a9b066926293699284"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:44.577389 27411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:44.614415 27411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:44.620185 27411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:44.620606 27411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:44.646221 27411 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:40263
I20260812 02:32:44.646235 27499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:40263 every 8 connection(s)
I20260812 02:32:44.648207 27411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:44.649006 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 27411
I20260812 02:32:44.649375 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:32:44.650377 12323 external_mini_cluster.cc:1428] Setting key bad75f59dacb412eef4aa60f1786a64b
I20260812 02:32:44.655951 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:44.676574 27500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Bootstrap starting.
Aug 12 02:32:44 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:44.681955 27500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:44.683782 27500 log.cc:824] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:44.688692 27500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: No bootstrap required, opened a new log
I20260812 02:32:44.692085 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:44.653983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.12.8.254:40263)
0812 02:32:44.657872 (+ 3889us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:44.657895 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:44.662035 (+ 4140us) server_negotiation.cc:407] Connection header received
0812 02:32:44.665808 (+ 3773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:44.665860 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:44.666756 (+ 896us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:44.667532 (+ 776us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:44.669415 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.671683 (+ 2268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.674007 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.675552 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.675842 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:44.681921 (+ 6079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:44.681943 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:44.681971 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:44.682040 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:44.686658 (+ 4618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:44.687660 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:44.687668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:44.687678 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:44.687891 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:44.688332 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:44.688336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:44.688339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:44.689126 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:44.689580 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:44.690037 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 02:32:44.690658 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3531,"thread_start_us":3247,"threads_started":1}
I20260812 02:32:44.700234 27500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } }
I20260812 02:32:44.700783 27500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:44.700896 27500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35af554318784f17a72635501ca5e8fb, State: Initialized, Role: FOLLOWER
I20260812 02:32:44.701659 27500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } }
I20260812 02:32:44.701952 27500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:44.702198 27500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:44.702445 27500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:44.705854 27500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } }
I20260812 02:32:44.706549 27500 leader_election.cc:302] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35af554318784f17a72635501ca5e8fb; no voters:
I20260812 02:32:44.707422 27500 leader_election.cc:288] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:44.707665 27505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:44.709893 27500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:44.711423 27505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 1 LEADER]: Becoming Leader. State: Replica: 35af554318784f17a72635501ca5e8fb, State: Running, Role: LEADER
I20260812 02:32:44.712431 27505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } }
I20260812 02:32:44.722445 27506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35af554318784f17a72635501ca5e8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } } }
I20260812 02:32:44.723196 27506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:44.724110 27507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35af554318784f17a72635501ca5e8fb. Latest consensus state: current_term: 1 leader_uuid: "35af554318784f17a72635501ca5e8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } } }
I20260812 02:32:44.724481 27507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:44.727293 27520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:44.727469 27500 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:32:44.747381 27520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:44.773991 27520 catalog_manager.cc:1347] Generated new cluster ID: 6a9375b4e3a74c28bf8a471fb3437462
I20260812 02:32:44.774176 27520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:44.790876 27520 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:44.792997 27520 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:44.804697 27520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Generated new TSK 0
I20260812 02:32:44.806227 27520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:32:45.827698 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:53507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40263
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:32:46 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786501966, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 02:32:46.281867 27555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:46.282290 27555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:46.282456 27555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:46.294332 27555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:46.294503 27555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:46.294680 27555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:32:46.310882 27555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.8.212:53507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:46.312888 27555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:46.314929 27555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:46.340965 27571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:46.341514 27568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:46.342398 27567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:46.351189 27555 server_base.cc:1034] running on GCE node
I20260812 02:32:46.352226 27555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:46.353850 27555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:46.355583 27555 hybrid_clock.cc:648] HybridClock initialized: now 1786501966355509 us; error 74 us; skew 500 ppm
I20260812 02:32:46.356171 27555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:46 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501966, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:46.361949 27555 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:32:46.367543 27555 webserver.cc:460] Webserver started at http://127.12.8.193:42083/ using document root <none> and password file <none>
I20260812 02:32:46.368723 27555 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:46.368911 27555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:46.369427 27555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:32:46 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501966, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:46.395377 27555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "0757c63cdc194341bb7b7c499d780cb3"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-vq2q"
server_key: "df68c31fe7381911e55b6c9da3228518"
server_key_iv: "d57150db04b949228bc444447e07c3fb"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:46.397219 27555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "0757c63cdc194341bb7b7c499d780cb3"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-vq2q"
server_key: "df68c31fe7381911e55b6c9da3228518"
server_key_iv: "d57150db04b949228bc444447e07c3fb"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:46.429327 27555 fs_manager.cc:691] Time spent creating directory manager: real 0.031s user 0.007s sys 0.010s
I20260812 02:32:46.469972 27580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:46.472618 27555 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.005s
I20260812 02:32:46.472954 27555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "0757c63cdc194341bb7b7c499d780cb3"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-vq2q"
server_key: "df68c31fe7381911e55b6c9da3228518"
server_key_iv: "d57150db04b949228bc444447e07c3fb"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:46.473407 27555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:46.508152 27555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:46.515830 27555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:46.516426 27555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:46.524534 27555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:46.533842 27555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:46.534123 27555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 02:32:46.534395 27555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:46.534566 27555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:32:46 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501966, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:46.762988 27555 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:46339
I20260812 02:32:46.763000 27715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:46339 every 8 connection(s)
I20260812 02:32:46.764595 27555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:46.774274 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 27555
I20260812 02:32:46.774503 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:46.776336 12323 external_mini_cluster.cc:1428] Setting key f542e935cd12333bcf7146b78908af32
Aug 12 02:32:46 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501966, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:46.835352 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:46.769731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:55499 (local address 127.12.8.254:40263)
0812 02:32:46.810473 (+ 40742us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:46.810480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:46.810506 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:32:46.816471 (+ 5965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:46.816478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:46.816583 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:46.816716 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:46.818994 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.820175 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.822441 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.823233 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.823378 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:46.827645 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:46.827652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:46.827656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:46.827697 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:46.831412 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:46.832405 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:46.832412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:46.832416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:46.832593 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:46.833149 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:46.833154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:46.833156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:46.833536 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:46.833666 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:46.834733 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0812 02:32:46.834976 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40595,"thread_start_us":43316,"threads_started":1}
I20260812 02:32:46.837734 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:46.768493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40263 (local address 127.12.8.193:55499)
0812 02:32:46.808014 (+ 39521us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:46.808092 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:46.814104 (+ 6012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:46.817072 (+ 2968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:46.817085 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:46.817832 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:46.818771 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:46.818801 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.820351 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.820356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:46.822279 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:46.822296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.823482 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.823491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:46.823779 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:46.824657 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:46.824687 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:46.827392 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:46.831605 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:46.831615 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:46.831633 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:46.832240 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:46.832741 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:46.832745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:46.832748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:46.833027 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:46.833763 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:46.833771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:46.834218 (+ 447us) client_negotiation.cc:769] Sending connection context
0812 02:32:46.835604 (+ 1386us) client_negotiation.cc:240] Negotiation successful
0812 02:32:46.836196 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39085,"thread_start_us":41892,"threads_started":1}
I20260812 02:32:46.840013 27716 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40263
I20260812 02:32:46.840378 27716 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:46.841399 27716 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
I20260812 02:32:46.844095 27453 ts_manager.cc:194] Registered new tserver with Master: 0757c63cdc194341bb7b7c499d780cb3 (127.12.8.193:46339)
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
I20260812 02:32:46.846329 27453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:55499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
W20260812 02:32:46.896135 12323 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.8.194: real 0.115s user 0.002s sys 0.026s
I20260812 02:32:46.897125 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:53507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40263
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:32:46 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501966, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
W20260812 02:32:47.388353 27723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:47.388681 27723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:47.388844 27723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:47.396296 27723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:47.396425 27723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:47.396600 27723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:32:47.406721 27723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.8.212:53507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:47.408057 27723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:47.409379 27723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:47.429993 27739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.430624 27737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.430694 27741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:47.433295 27723 server_base.cc:1034] running on GCE node
I20260812 02:32:47.433907 27723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:47.434999 27723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:47.436262 27723 hybrid_clock.cc:648] HybridClock initialized: now 1786501967436198 us; error 48 us; skew 500 ppm
I20260812 02:32:47.436633 27723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:47 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501967, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:47.440065 27723 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:32:47.441713 27723 webserver.cc:460] Webserver started at http://127.12.8.194:36207/ using document root <none> and password file <none>
I20260812 02:32:47.442422 27723 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:47.442555 27723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:47.442886 27723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:32:47 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501967, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:47.461397 27723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "d418d316cfa94ceaabe354287bfe46ed"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-vq2q"
server_key: "7d7db6920cec3c2397f27680e6a0681a"
server_key_iv: "f09b715f0d1b268ace4a37cda5642cb5"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:47.462318 27723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "d418d316cfa94ceaabe354287bfe46ed"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-vq2q"
server_key: "7d7db6920cec3c2397f27680e6a0681a"
server_key_iv: "f09b715f0d1b268ace4a37cda5642cb5"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:47.467962 27723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 02:32:47.482746 27749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.484319 27723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:47.484570 27723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "d418d316cfa94ceaabe354287bfe46ed"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-vq2q"
server_key: "7d7db6920cec3c2397f27680e6a0681a"
server_key_iv: "f09b715f0d1b268ace4a37cda5642cb5"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:47.484802 27723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:47.506713 27723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:47.510833 27723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:47.511121 27723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:47.512197 27723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:47.513823 27723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:47.513914 27723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.514036 27723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:47.514094 27723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.565202 27723 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:39609
I20260812 02:32:47.565207 27894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:39609 every 8 connection(s)
I20260812 02:32:47.566656 27723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:47.575110 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 27723
I20260812 02:32:47.575304 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:47.575917 12323 external_mini_cluster.cc:1428] Setting key 57579cb826c61609bdd85caacc8a4230
Aug 12 02:32:47 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501967, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
I20260812 02:32:47.621906 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.570428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40263 (local address 127.12.8.194:43569)
0812 02:32:47.588045 (+ 17617us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:47.588113 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:47.590611 (+ 2498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:47.602304 (+ 11693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:47.602318 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:47.602990 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:47.603935 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:47.603966 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.605887 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.605894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:47.608005 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:47.608025 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.609370 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.609378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:47.609680 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:47.610493 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:47.610526 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:47.613629 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:47.617212 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:47.617222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:47.617240 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:47.617785 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:47.618198 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:47.618204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:47.618208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:47.618451 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:47.619188 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:47.619196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:47.619629 (+ 433us) client_negotiation.cc:769] Sending connection context
0812 02:32:47.619932 (+ 303us) client_negotiation.cc:240] Negotiation successful
0812 02:32:47.620474 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17260,"thread_start_us":17014,"threads_started":1}
I20260812 02:32:47.622927 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.569743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:43569 (local address 127.12.8.254:40263)
0812 02:32:47.586179 (+ 16436us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:47.586186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:47.591022 (+ 4836us) server_negotiation.cc:407] Connection header received
0812 02:32:47.591100 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:47.591104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:47.591188 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:47.591287 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:47.604281 (+ 12994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.605562 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.608225 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.608865 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.608991 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:47.613946 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:47.613953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:47.613958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:47.614000 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:47.617044 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:47.617917 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:47.617921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:47.617924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:47.618088 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:47.618567 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:47.618572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:47.618575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:47.618969 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:47.619113 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:47.622370 (+ 3257us) server_negotiation.cc:299] Negotiation successful
0812 02:32:47.622589 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16283,"thread_start_us":16439,"threads_started":1}
I20260812 02:32:47.625048 27896 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40263
I20260812 02:32:47.625394 27896 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:47.626263 27896 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
I20260812 02:32:47.628089 27453 ts_manager.cc:194] Registered new tserver with Master: d418d316cfa94ceaabe354287bfe46ed (127.12.8.194:39609)
I20260812 02:32:47.629195 27453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:43569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:32:47.661530 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:53507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40263
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:32:47.850762 27716 heartbeater.cc:503] Master 127.12.8.254:40263 was elected leader, sending a full tablet report...
W20260812 02:32:48.105810 27904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:48.106221 27904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:48.106345 27904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:48.115056 27904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:48.115227 27904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:48.115370 27904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:32:48.126832 27904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.8.212:53507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:48.128654 27904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:48.130447 27904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:48.146205 27909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:48.146382 27910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:48.146351 27912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:48.147846 27904 server_base.cc:1034] running on GCE node
I20260812 02:32:48.148610 27904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:48.149917 27904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:48.151160 27904 hybrid_clock.cc:648] HybridClock initialized: now 1786501968151087 us; error 62 us; skew 500 ppm
I20260812 02:32:48.151561 27904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:48 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501968, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:48.155856 27904 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:32:48.157821 27904 webserver.cc:460] Webserver started at http://127.12.8.195:35231/ using document root <none> and password file <none>
I20260812 02:32:48.158643 27904 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:48.158761 27904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:48.159070 27904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:32:48 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501968, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:48.177976 27904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "0953fdae3ce746fc8a1b8f67312df383"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-vq2q"
server_key: "7cab925c525cbe6ce198688a324613e2"
server_key_iv: "e95c416b660168232ef2dc638363f2ef"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:48.179252 27904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "0953fdae3ce746fc8a1b8f67312df383"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-vq2q"
server_key: "7cab925c525cbe6ce198688a324613e2"
server_key_iv: "e95c416b660168232ef2dc638363f2ef"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:48.187269 27904 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260812 02:32:48.209585 27921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.211323 27904 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:48.211527 27904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "0953fdae3ce746fc8a1b8f67312df383"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-vq2q"
server_key: "7cab925c525cbe6ce198688a324613e2"
server_key_iv: "e95c416b660168232ef2dc638363f2ef"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:48.211772 27904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:48.238016 27904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:48.242543 27904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:48.242851 27904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:48.243963 27904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:48.245644 27904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:48.245743 27904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.245886 27904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:48.245973 27904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.283298 27904 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:34381
I20260812 02:32:48.283444 28059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:34381 every 8 connection(s)
I20260812 02:32:48.284739 27904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:48.293516 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 27904
I20260812 02:32:48.293689 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:48.294189 12323 external_mini_cluster.cc:1428] Setting key 5681b87678769446cbb242a0186c39c8
Aug 12 02:32:48 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501968, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:48.311699 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.288234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:42717 (local address 127.12.8.254:40263)
0812 02:32:48.288875 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:48.288882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:48.290267 (+ 1385us) server_negotiation.cc:407] Connection header received
0812 02:32:48.292668 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:48.292673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:48.292755 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:48.292883 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:48.295359 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.296387 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.298994 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.299446 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.299550 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:48.304490 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:48.304496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:48.304500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:48.304546 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:48.307378 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:48.308515 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:48.308521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:48.308524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:48.308705 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:48.309378 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:48.309383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:48.309386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:48.309805 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:48.309953 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:48.311157 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0812 02:32:48.311394 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":310,"threads_started":1}
I20260812 02:32:48.313078 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.288765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40263 (local address 127.12.8.195:42717)
0812 02:32:48.290019 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:48.290102 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:48.292260 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:48.293352 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:48.293365 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:48.294067 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:48.294984 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:48.295013 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.296629 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.296635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:48.298767 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:48.298802 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.299768 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.299775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:48.300083 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:48.301096 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:48.301134 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:48.304102 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:48.307604 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:48.307615 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:48.307635 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:48.308293 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:48.308890 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:48.308895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:48.308898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:48.309238 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:48.309980 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:48.309991 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:48.310616 (+ 625us) client_negotiation.cc:769] Sending connection context
0812 02:32:48.311010 (+ 394us) client_negotiation.cc:240] Negotiation successful
0812 02:32:48.311673 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":483,"threads_started":1}
I20260812 02:32:48.314893 28060 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40263
I20260812 02:32:48.315295 28060 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:48.316316 28060 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
I20260812 02:32:48.318161 27453 ts_manager.cc:194] Registered new tserver with Master: 0953fdae3ce746fc8a1b8f67312df383 (127.12.8.195:34381)
I20260812 02:32:48.319103 27453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:42717
I20260812 02:32:48.324335 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:32:48.326412 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 27555
Aug 12 02:32:48 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:32:48.345779 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.327524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.12.8.193:46339)
0812 02:32:48.328164 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:48.328171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:48.328203 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:32:48.328405 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:48.328412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:48.328635 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:48.328867 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:48.330809 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.331887 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.333474 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.334059 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.334166 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:48.339149 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:48.339164 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:48.339180 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:48.339223 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:48.342281 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:48.343291 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:48.343298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:48.343300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:48.343484 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:48.343902 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:48.343905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:48.343907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:48.344271 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:48.344555 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:48.345076 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 02:32:48.345432 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":308,"threads_started":1}
I20260812 02:32:48.347362 27708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:48.443063 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 27555
I20260812 02:32:48.464916 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 27723
Aug 12 02:32:48 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:32:48.490774 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.471539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.12.8.194:39609)
0812 02:32:48.472224 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:48.472233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:48.472272 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:32:48.472359 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:48.472367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:48.472716 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:48.472921 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:48.474321 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.475467 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.477102 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.477945 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.478114 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:48.483007 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:48.483027 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:48.483048 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:48.483102 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:48.486837 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:48.487662 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:48.487675 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:48.487679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:48.487969 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:48.488514 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:48.488519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:48.488522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:48.489108 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:48.489649 (+ 541us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:48.489933 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 02:32:48.490368 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":331,"threads_started":1}
I20260812 02:32:48.492443 27890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:48.579433 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 27723
I20260812 02:32:48.604089 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 27904
Aug 12 02:32:48 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:32:48.624505 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.605477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.12.8.195:34381)
0812 02:32:48.606267 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:48.606275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:48.606306 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:32:48.606402 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:48.606409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:48.606630 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:48.606772 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:48.608431 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.609548 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.611096 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:48.611719 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:48.611831 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:48.617554 (+ 5723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:48.617569 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:48.617584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:48.617625 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:48.620804 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:48.622059 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:48.622067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:48.622069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:48.622232 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:48.622744 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:48.622748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:48.622750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:48.623166 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:48.623461 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:48.623838 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 02:32:48.624082 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":407,"threads_started":1}
I20260812 02:32:48.626370 28055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:48.724664 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 27904
I20260812 02:32:48.745401 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 27411
I20260812 02:32:48.746919 27495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:48.862057 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 27411
I20260812 02:32:48.891945 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40263
--webserver_interface=127.12.8.254
--webserver_port=37693
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40263
--ranger_config_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:53507/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.8.212:53507/kms with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:49.320873 28077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:49.321388 28077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:49.321535 28077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:49.329794 28077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:49.329957 28077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:49.330008 28077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:49.330099 28077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:49.330168 28077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:49.342649 28077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.8.212:53507/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40263
--ranger_config_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=37693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:49.344718 28077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:49.346498 28077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:49.361001 28084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:49.361164 28085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:49.361232 28087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:49.362177 28077 server_base.cc:1034] running on GCE node
I20260812 02:32:49.363323 28077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:49.365433 28077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:49.366717 28077 hybrid_clock.cc:648] HybridClock initialized: now 1786501969366650 us; error 49 us; skew 500 ppm
I20260812 02:32:49.367224 28077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:49 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501969, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:49.371418 28077 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:32:49.373579 28077 webserver.cc:460] Webserver started at http://127.12.8.254:37693/ using document root <none> and password file <none>
I20260812 02:32:49.374462 28077 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:49.374631 28077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:32:49 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501969, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:49.461488 28077 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:32:49.465582 28096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:49.467515 28077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:49.467810 28077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "35af554318784f17a72635501ca5e8fb"
format_stamp: "Formatted at 2026-08-12 02:32:44 on dist-test-slave-vq2q"
server_key: "90fd7573f0e16b04c5608c253dac8c61"
server_key_iv: "b9ab24d5699834a9b066926293699284"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:49.469033 28077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:49.485652 28077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:49.490846 28077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:49.491192 28077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:49.513979 28077 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:40263
I20260812 02:32:49.514067 28162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:40263 every 8 connection(s)
I20260812 02:32:49.516119 28077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:49.520218 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 28077
I20260812 02:32:49.521353 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:53507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:46339
--local_ip_for_outbound_sockets=127.12.8.193
--tserver_master_addrs=127.12.8.254:40263
--webserver_port=42083
--webserver_interface=127.12.8.193
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.8.212:53507/kms with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:32:49.524000 28164 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:32:49.527506 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Bootstrap starting.
I20260812 02:32:49.547186 28164 log.cc:824] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:49.556665 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:32:49.557210 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Bootstrap complete.
I20260812 02:32:49.566380 28164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } }
I20260812 02:32:49.567273 28164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35af554318784f17a72635501ca5e8fb, State: Initialized, Role: FOLLOWER
I20260812 02:32:49.567955 28164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } }
I20260812 02:32:49.568182 28164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:49.568295 28164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:49.568461 28164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:32:49.572531 28164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } }
I20260812 02:32:49.573103 28164 leader_election.cc:302] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35af554318784f17a72635501ca5e8fb; no voters:
I20260812 02:32:49.573931 28164 leader_election.cc:288] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:32:49.574311 28169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:32:49.575240 28169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [term 2 LEADER]: Becoming Leader. State: Replica: 35af554318784f17a72635501ca5e8fb, State: Running, Role: LEADER
I20260812 02:32:49.576126 28164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:49.575953 28169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } }
I20260812 02:32:49.587172 28173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35af554318784f17a72635501ca5e8fb. Latest consensus state: current_term: 2 leader_uuid: "35af554318784f17a72635501ca5e8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } } }
I20260812 02:32:49.587687 28173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:49.590780 28172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "35af554318784f17a72635501ca5e8fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35af554318784f17a72635501ca5e8fb" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40263 } } }
I20260812 02:32:49.591105 28172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:49.591439 28190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:49.591562 28164 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:32:49.607538 28190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:49.609511 28190 catalog_manager.cc:1259] Loaded cluster ID: 6a9375b4e3a74c28bf8a471fb3437462
I20260812 02:32:49.609637 28190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:49.614459 28190 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:49.616271 28190 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Loaded TSK: 0
I20260812 02:32:49.618153 28190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:32:50.102934 28167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:50.103353 28167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:50.103503 28167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:50.114836 28167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:50.115072 28167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:50.115317 28167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:32:50.132814 28167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.8.212:53507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:46339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=42083
--tserver_master_addrs=127.12.8.254:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:50.134852 28167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:50.136574 28167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:50.159981 28215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:50.161053 28219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:50.166790 28222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:50.167470 28167 server_base.cc:1034] running on GCE node
I20260812 02:32:50.168519 28167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:50.170197 28167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:50.171449 28167 hybrid_clock.cc:648] HybridClock initialized: now 1786501970171365 us; error 70 us; skew 500 ppm
I20260812 02:32:50.171948 28167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:50 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:50.176146 28167 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:32:50.186160 28167 webserver.cc:460] Webserver started at http://127.12.8.193:42083/ using document root <none> and password file <none>
I20260812 02:32:50.187345 28167 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:50.187546 28167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:32:50 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:50.227202 28167 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.007s sys 0.007s
I20260812 02:32:50.241089 28235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.244099 28167 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.005s sys 0.004s
I20260812 02:32:50.244454 28167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "0757c63cdc194341bb7b7c499d780cb3"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-vq2q"
server_key: "df68c31fe7381911e55b6c9da3228518"
server_key_iv: "d57150db04b949228bc444447e07c3fb"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:50.246186 28167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:50.271543 28167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:50.277302 28167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:50.277655 28167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:50.279151 28167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:50.281014 28167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:50.281123 28167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.281266 28167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:50.281347 28167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.358297 28167 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:46339
I20260812 02:32:50.358445 28394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:46339 every 8 connection(s)
I20260812 02:32:50.360849 28167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:50.370909 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 28167
I20260812 02:32:50.371845 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:53507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:39609
--local_ip_for_outbound_sockets=127.12.8.194
--tserver_master_addrs=127.12.8.254:40263
--webserver_port=36207
--webserver_interface=127.12.8.194
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.8.212:53507/kms with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:32:50 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:50.414877 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.366565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:38241 (local address 127.12.8.254:40263)
0812 02:32:50.373958 (+ 7393us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.373987 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.374837 (+ 850us) server_negotiation.cc:407] Connection header received
0812 02:32:50.383188 (+ 8351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.383213 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.383974 (+ 761us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.384543 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:50.387982 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.390184 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.393079 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.394229 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.394473 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.402733 (+ 8260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:50.402750 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:50.402778 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:50.402850 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:50.408147 (+ 5297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.409308 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.409321 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.409329 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.409627 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.410308 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.410314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.410318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.411231 (+ 913us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:50.411475 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.412592 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.413063 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6970,"thread_start_us":6668,"threads_started":1}
I20260812 02:32:50.414889 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.366611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40263 (local address 127.12.8.193:38241)
0812 02:32:50.374565 (+ 7954us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.374640 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.381827 (+ 7187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.385594 (+ 3767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.385609 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.386480 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:50.387553 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.387594 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.390502 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.390510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.392786 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.392815 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.395048 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.395057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.395413 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.396867 (+ 1454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:50.396917 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:50.402200 (+ 5283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:50.408413 (+ 6213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:50.408424 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:50.408441 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:50.409044 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:50.409799 (+ 755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:50.409804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:50.409807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:50.410140 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:50.411470 (+ 1330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:50.411480 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:50.411949 (+ 469us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.412307 (+ 358us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.413087 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7550,"thread_start_us":9101,"threads_started":1}
I20260812 02:32:50.421005 28396 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40263
I20260812 02:32:50.421514 28396 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:50.422760 28396 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
W20260812 02:32:50.427739 28396 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:40263 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:32:50.428254 28396 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:50.428642 28396 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
I20260812 02:32:50.430176 28396 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:50.430608 28396 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
W20260812 02:32:50.432031 28396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:32:50.812106 28402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:50.812501 28402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:50.812634 28402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:50.820953 28402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:50.821100 28402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:50.821290 28402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:32:50.835227 28402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.8.212:53507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:39609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=36207
--tserver_master_addrs=127.12.8.254:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:50.837081 28402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:50.838948 28402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:50.862010 28421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:50.864163 28422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:50.867580 28426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:50.868112 28402 server_base.cc:1034] running on GCE node
I20260812 02:32:50.869349 28402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:50.878571 28402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:50.879936 28402 hybrid_clock.cc:648] HybridClock initialized: now 1786501970879866 us; error 79 us; skew 500 ppm
I20260812 02:32:50.880478 28402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:50 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:50.885243 28402 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:32:50.893940 28402 webserver.cc:460] Webserver started at http://127.12.8.194:36207/ using document root <none> and password file <none>
I20260812 02:32:50.894999 28402 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:50.895200 28402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:32:50 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:50.931635 28402 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.008s sys 0.004s
I20260812 02:32:50.935925 28437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.938043 28402 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:32:50.938338 28402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "d418d316cfa94ceaabe354287bfe46ed"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-vq2q"
server_key: "7d7db6920cec3c2397f27680e6a0681a"
server_key_iv: "f09b715f0d1b268ace4a37cda5642cb5"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:50.939905 28402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 02:32:50 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:50.965756 28402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:50.971375 28402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:50.971721 28402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:50.973105 28402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:50.983407 28402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:50.983554 28402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.983693 28402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:50.983791 28402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:51.050468 28402 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:39609
I20260812 02:32:51.050518 28579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:39609 every 8 connection(s)
I20260812 02:32:51.052477 28402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:51.057991 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 28402
I20260812 02:32:51.058808 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.212:53507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:34381
--local_ip_for_outbound_sockets=127.12.8.195
--tserver_master_addrs=127.12.8.254:40263
--webserver_port=35231
--webserver_interface=127.12.8.195
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.8.212:53507/kms with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:32:51 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:51.097563 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:51.058264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:49473 (local address 127.12.8.254:40263)
0812 02:32:51.073645 (+ 15381us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:51.073651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:51.073675 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:32:51.073754 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:51.073760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:51.073850 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:51.073977 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:51.076351 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:51.077227 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:51.079057 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:51.079695 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:51.081399 (+ 1704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:51.088172 (+ 6773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:51.088181 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:51.088185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:51.088230 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:51.091836 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:51.092959 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:51.092978 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:51.092981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:51.093249 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:51.093807 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:51.093812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:51.093815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:51.094192 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:51.094365 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:51.096828 (+ 2463us) server_negotiation.cc:299] Negotiation successful
0812 02:32:51.097112 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15227,"thread_start_us":15034,"threads_started":1}
I20260812 02:32:51.097596 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:51.057499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40263 (local address 127.12.8.194:49473)
0812 02:32:51.067442 (+ 9943us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:51.067529 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:51.070296 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:51.074481 (+ 4185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:51.074494 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:51.075279 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:51.076104 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:51.076135 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:51.077423 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:51.077428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:51.078900 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:51.078915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:51.081717 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:51.081722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:51.081945 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:51.085170 (+ 3225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:51.085225 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:51.087823 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:51.092056 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:51.092066 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:51.092086 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:51.092803 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:51.093392 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:51.093397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:51.093400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:51.093672 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:51.094466 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:51.094474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:51.095004 (+ 530us) client_negotiation.cc:769] Sending connection context
0812 02:32:51.095347 (+ 343us) client_negotiation.cc:240] Negotiation successful
0812 02:32:51.096018 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9521,"thread_start_us":13143,"threads_started":1}
I20260812 02:32:51.099761 28581 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40263
I20260812 02:32:51.100183 28581 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:51.101275 28581 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
I20260812 02:32:51.103710 28120 ts_manager.cc:194] Registered new tserver with Master: d418d316cfa94ceaabe354287bfe46ed (127.12.8.194:39609)
I20260812 02:32:51.106063 28120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:49473
Aug 12 02:32:51 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:51.432194 28396 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:51.432690 28396 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
I20260812 02:32:51.434088 28120 ts_manager.cc:194] Registered new tserver with Master: 0757c63cdc194341bb7b7c499d780cb3 (127.12.8.193:46339)
I20260812 02:32:51.435292 28120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:38241
W20260812 02:32:51.554508 28586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:51.554826 28586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:51.554994 28586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:51.562598 28586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:51.562722 28586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:51.562906 28586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:32:51.573298 28586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.8.212:53507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:34381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=35231
--tserver_master_addrs=127.12.8.254:40263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:51.574729 28586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:51.576113 28586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:51.593436 28594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:51.605355 28595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:51.606240 28597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:51.606824 28586 server_base.cc:1034] running on GCE node
I20260812 02:32:51.607664 28586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:51.609092 28586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:51.610373 28586 hybrid_clock.cc:648] HybridClock initialized: now 1786501971610304 us; error 48 us; skew 500 ppm
I20260812 02:32:51.610762 28586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:51 dist-test-slave-vq2q krb5kdc[26851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501971, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:51.614998 28586 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:32:51.616930 28586 webserver.cc:460] Webserver started at http://127.12.8.195:35231/ using document root <none> and password file <none>
I20260812 02:32:51.617913 28586 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:51.618157 28586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:32:51 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501971, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:51.646739 28586 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.003s
I20260812 02:32:51.650974 28605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:51.654027 28586 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 02:32:51.654234 28586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "0953fdae3ce746fc8a1b8f67312df383"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-vq2q"
server_key: "7cab925c525cbe6ce198688a324613e2"
server_key_iv: "e95c416b660168232ef2dc638363f2ef"
server_key_version: "kuduclusterkey@0"
I20260812 02:32:51.655606 28586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:51.690804 28586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:51.696115 28586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:51.696429 28586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:51.700181 28586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:51.702098 28586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:51.702214 28586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:51.702322 28586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:51.702394 28586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:32:51 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for HTTP/127.12.8.212@KRBTEST.COM
I20260812 02:32:51.970681 28586 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:34381
I20260812 02:32:51.970893 28746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:34381 every 8 connection(s)
I20260812 02:32:51.972474 28586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:51.979935 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 28586
Aug 12 02:32:52 dist-test-slave-vq2q krb5kdc[26851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501971, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:52.018471 28751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:51.977288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:48817 (local address 127.12.8.254:40263)
0812 02:32:51.997657 (+ 20369us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:51.997662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:51.997688 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:32:51.997755 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:51.997759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:51.997834 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:51.997964 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:52.000362 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.001591 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.004569 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.005319 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.005468 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.010665 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:52.010672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:52.010676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:52.010746 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:52.014611 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.015642 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.015647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.015649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.015789 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.016253 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.016258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.016262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.016655 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:52.016819 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.017818 (+ 999us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.018053 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20233,"thread_start_us":20062,"threads_started":1}
I20260812 02:32:52.020089 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:51.976504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40263 (local address 127.12.8.195:48817)
0812 02:32:51.981778 (+ 5274us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:51.981844 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:51.984356 (+ 2512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:51.998426 (+ 14070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:51.998443 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:51.999184 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:52.000106 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.000136 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.002144 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.002150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.004372 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.004390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.005863 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.005870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.006189 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.007193 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:52.007249 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:52.010373 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:52.014791 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:52.014802 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:52.014822 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:52.015497 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:52.015926 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:52.015930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:52.015932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:52.016131 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:52.016817 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:52.016827 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:52.017344 (+ 517us) client_negotiation.cc:769] Sending connection context
0812 02:32:52.017695 (+ 351us) client_negotiation.cc:240] Negotiation successful
0812 02:32:52.018281 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4894,"thread_start_us":11463,"threads_started":1}
I20260812 02:32:52.021952 28748 heartbeater.cc:347] Connected to a master server at 127.12.8.254:40263
I20260812 02:32:52.022343 28748 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:52.023350 28748 heartbeater.cc:511] Master 127.12.8.254:40263 requested a full tablet report, sending...
I20260812 02:32:52.025334 28120 ts_manager.cc:194] Registered new tserver with Master: 0953fdae3ce746fc8a1b8f67312df383 (127.12.8.195:34381)
I20260812 02:32:52.026603 28120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:48817
I20260812 02:32:52.027791 28753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:51.981539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.12.8.254:40263)
0812 02:32:52.009602 (+ 28063us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.009609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.009630 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:32:52.010676 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.010681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.010787 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.011857 (+ 1070us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:52.013005 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.013977 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.015619 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.016319 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.016445 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.019612 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:52.019617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:52.019620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:52.019661 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:52.023255 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.024607 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.024614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.024618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.024816 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.025556 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.025562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.025566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.025917 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:52.026229 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.026557 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.026799 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27941,"thread_start_us":28158,"threads_started":1}
I20260812 02:32:52.030692 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:32:52.048903 28753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.034204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.12.8.254:40263)
0812 02:32:52.034411 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.034418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.034935 (+ 517us) server_negotiation.cc:407] Connection header received
0812 02:32:52.035142 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.035146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.035210 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.035294 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:52.037308 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.038028 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.040003 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.040439 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.040534 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.043210 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:52.043215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:52.043218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:52.043255 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:52.045975 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.046893 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.046897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.046900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.047120 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.047512 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.047516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.047518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.047818 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:52.047999 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.048430 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.048609 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:32:52.054008 28120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:32:52.061549 28120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:52.096105 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.076302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:39609 (local address 127.0.0.1:53008)
0812 02:32:52.086146 (+ 9844us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.086164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:52.086282 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:52.088550 (+ 2268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:52.088557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:52.088587 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:52.088999 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.089015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.090623 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.090628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.093233 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.093250 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.095172 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.095177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.095326 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.095366 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 02:32:52.095495 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 02:32:52.095705 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9723,"thread_start_us":9591,"threads_started":1}
I20260812 02:32:52.096094 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.076755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.12.8.194:39609)
0812 02:32:52.087770 (+ 11015us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.087777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.087803 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:32:52.087890 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.087896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.088092 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.088546 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:52.089234 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.090486 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.093445 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.094943 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.095071 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.095309 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.095578 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.095758 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10858,"thread_start_us":12002,"threads_started":1}
I20260812 02:32:52.098229 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.075741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46339 (local address 127.0.0.1:56456)
0812 02:32:52.082684 (+ 6943us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.082728 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:52.082990 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:52.091420 (+ 8430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:52.091425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:52.091453 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:52.091848 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.091857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.093292 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.093298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.095455 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.095473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.097413 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.097421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.097586 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.097618 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:32:52.097719 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 02:32:52.097841 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6665,"mutex_wait_us":45,"thread_start_us":6501,"threads_started":1}
I20260812 02:32:52.098254 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.076121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.12.8.193:46339)
0812 02:32:52.090862 (+ 14741us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.090870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.090896 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:32:52.091000 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.091007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.091200 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.091418 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:52.092030 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.093112 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.095647 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.097169 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.097361 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.097574 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.097777 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.097933 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14574,"thread_start_us":13668,"threads_started":1}
I20260812 02:32:52.100442 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.088433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:34381 (local address 127.0.0.1:33880)
0812 02:32:52.089621 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.089639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:52.089748 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:52.091011 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:52.091016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:52.091041 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:52.091509 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.091523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.093480 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.093485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.097169 (+ 3684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.097209 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.099609 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.099616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.099788 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.099833 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 02:32:52.099933 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 02:32:52.100064 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":4,"thread_start_us":907,"threads_started":1}
I20260812 02:32:52.100559 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.083524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.12.8.195:34381)
0812 02:32:52.090528 (+ 7004us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.090535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.090563 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:32:52.090636 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.090641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.090814 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.091002 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:52.091716 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.093319 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.097402 (+ 4083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.099118 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.099241 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.099450 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.100010 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.100185 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6856,"thread_start_us":6667,"threads_started":1}
I20260812 02:32:52.100517 28497 tablet_service.cc:1467] Processing CreateTablet for tablet f0124fb3ee3442d0b7710b674d3ab054 (DEFAULT_TABLE table=test-table [id=0ebd625c7f0a4d70b3ebceb2e64060d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:52.102233 28497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0124fb3ee3442d0b7710b674d3ab054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:52.102093 28302 tablet_service.cc:1467] Processing CreateTablet for tablet f0124fb3ee3442d0b7710b674d3ab054 (DEFAULT_TABLE table=test-table [id=0ebd625c7f0a4d70b3ebceb2e64060d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:52.103576 28302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0124fb3ee3442d0b7710b674d3ab054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:52.104614 28662 tablet_service.cc:1467] Processing CreateTablet for tablet f0124fb3ee3442d0b7710b674d3ab054 (DEFAULT_TABLE table=test-table [id=0ebd625c7f0a4d70b3ebceb2e64060d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:52.106048 28662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0124fb3ee3442d0b7710b674d3ab054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:52.111184 28581 heartbeater.cc:503] Master 127.12.8.254:40263 was elected leader, sending a full tablet report...
I20260812 02:32:52.115263 28782 tablet_bootstrap.cc:492] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: Bootstrap starting.
I20260812 02:32:52.118501 28782 tablet_bootstrap.cc:654] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:52.118919 28783 tablet_bootstrap.cc:492] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3: Bootstrap starting.
I20260812 02:32:52.119992 28782 log.cc:824] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:52.120334 28786 tablet_bootstrap.cc:492] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Bootstrap starting.
I20260812 02:32:52.122668 28783 tablet_bootstrap.cc:654] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:52.123241 28786 tablet_bootstrap.cc:654] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:52.123989 28783 log.cc:824] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:52.124183 28786 log.cc:824] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:52.126904 28786 tablet_bootstrap.cc:492] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: No bootstrap required, opened a new log
I20260812 02:32:52.126997 28783 tablet_bootstrap.cc:492] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3: No bootstrap required, opened a new log
I20260812 02:32:52.127007 28782 tablet_bootstrap.cc:492] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: No bootstrap required, opened a new log
I20260812 02:32:52.127177 28786 ts_tablet_manager.cc:1397] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:32:52.127264 28782 ts_tablet_manager.cc:1397] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260812 02:32:52.127261 28783 ts_tablet_manager.cc:1397] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 02:32:52.135128 28786 raft_consensus.cc:348] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
I20260812 02:32:52.135509 28786 raft_consensus.cc:374] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:52.135609 28786 raft_consensus.cc:729] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0953fdae3ce746fc8a1b8f67312df383, State: Initialized, Role: FOLLOWER
I20260812 02:32:52.136281 28786 consensus_queue.cc:260] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
I20260812 02:32:52.137472 28748 heartbeater.cc:503] Master 127.12.8.254:40263 was elected leader, sending a full tablet report...
I20260812 02:32:52.138001 28786 ts_tablet_manager.cc:1428] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260812 02:32:52.138060 28783 raft_consensus.cc:348] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
I20260812 02:32:52.138491 28783 raft_consensus.cc:374] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:52.138592 28783 raft_consensus.cc:729] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0757c63cdc194341bb7b7c499d780cb3, State: Initialized, Role: FOLLOWER
I20260812 02:32:52.139204 28783 consensus_queue.cc:260] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
I20260812 02:32:52.139204 28782 raft_consensus.cc:348] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
I20260812 02:32:52.139614 28782 raft_consensus.cc:374] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:52.139724 28782 raft_consensus.cc:729] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d418d316cfa94ceaabe354287bfe46ed, State: Initialized, Role: FOLLOWER
I20260812 02:32:52.140519 28396 heartbeater.cc:503] Master 127.12.8.254:40263 was elected leader, sending a full tablet report...
I20260812 02:32:52.140424 28782 consensus_queue.cc:260] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
I20260812 02:32:52.141232 28783 ts_tablet_manager.cc:1428] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260812 02:32:52.148310 28782 ts_tablet_manager.cc:1428] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260812 02:32:52.203996 28792 raft_consensus.cc:482] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:52.204329 28792 raft_consensus.cc:504] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
I20260812 02:32:52.206403 28792 leader_election.cc:288] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0953fdae3ce746fc8a1b8f67312df383 (127.12.8.195:34381), d418d316cfa94ceaabe354287bfe46ed (127.12.8.194:39609)
I20260812 02:32:52.216463 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.207011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:48449 (local address 127.12.8.195:34381)
0812 02:32:52.207185 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.207190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.207996 (+ 806us) server_negotiation.cc:407] Connection header received
0812 02:32:52.208180 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.208186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.208264 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.208371 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:52.209387 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.210542 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.213721 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.215141 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.215357 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.215440 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.216025 (+ 585us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.216181 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:32:52.216468 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.206942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:34433 (local address 127.12.8.194:39609)
0812 02:32:52.207162 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.207169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.207952 (+ 783us) server_negotiation.cc:407] Connection header received
0812 02:32:52.208176 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.208182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.208263 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.208382 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:52.209414 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.210521 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.213727 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.215141 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.215358 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.215440 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.216025 (+ 585us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.216181 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:32:52.216630 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.206651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:39609 (local address 127.12.8.193:34433)
0812 02:32:52.207784 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.207812 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:52.207983 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:52.208439 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:52.208445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:52.208480 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:52.209165 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.209207 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.210701 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.210708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.213484 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.213507 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.215529 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.215534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.215757 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.215827 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 02:32:52.215941 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 02:32:52.216125 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":389,"thread_start_us":732,"threads_started":1}
I20260812 02:32:52.216922 28795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.206887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:34381 (local address 127.12.8.193:48449)
0812 02:32:52.207868 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.207883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:52.208037 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:52.208439 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:52.208445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:52.208492 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:52.209121 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.209138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.210701 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.210707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.213484 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:52.213507 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.215500 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.215512 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:52.215773 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.215832 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 02:32:52.215941 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 02:32:52.216125 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":290,"spinlock_wait_cycles":60416,"thread_start_us":595,"threads_started":1}
I20260812 02:32:52.217999 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0124fb3ee3442d0b7710b674d3ab054" candidate_uuid: "0757c63cdc194341bb7b7c499d780cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d418d316cfa94ceaabe354287bfe46ed" is_pre_election: true
I20260812 02:32:52.217999 28692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0124fb3ee3442d0b7710b674d3ab054" candidate_uuid: "0757c63cdc194341bb7b7c499d780cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0953fdae3ce746fc8a1b8f67312df383" is_pre_election: true
I20260812 02:32:52.218645 28692 raft_consensus.cc:2454] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0757c63cdc194341bb7b7c499d780cb3 in term 0.
I20260812 02:32:52.218644 28529 raft_consensus.cc:2454] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0757c63cdc194341bb7b7c499d780cb3 in term 0.
I20260812 02:32:52.219430 28238 leader_election.cc:302] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0757c63cdc194341bb7b7c499d780cb3, d418d316cfa94ceaabe354287bfe46ed; no voters:
I20260812 02:32:52.219900 28792 raft_consensus.cc:2781] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:52.220053 28792 raft_consensus.cc:482] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:52.220142 28792 raft_consensus.cc:3037] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:52.222484 28792 raft_consensus.cc:504] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
I20260812 02:32:52.223135 28792 leader_election.cc:288] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 0953fdae3ce746fc8a1b8f67312df383 (127.12.8.195:34381), d418d316cfa94ceaabe354287bfe46ed (127.12.8.194:39609)
I20260812 02:32:52.223850 28692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0124fb3ee3442d0b7710b674d3ab054" candidate_uuid: "0757c63cdc194341bb7b7c499d780cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0953fdae3ce746fc8a1b8f67312df383"
I20260812 02:32:52.223883 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0124fb3ee3442d0b7710b674d3ab054" candidate_uuid: "0757c63cdc194341bb7b7c499d780cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d418d316cfa94ceaabe354287bfe46ed"
I20260812 02:32:52.224105 28692 raft_consensus.cc:3037] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:52.224105 28529 raft_consensus.cc:3037] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:52.227475 28529 raft_consensus.cc:2454] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0757c63cdc194341bb7b7c499d780cb3 in term 1.
I20260812 02:32:52.227475 28692 raft_consensus.cc:2454] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0757c63cdc194341bb7b7c499d780cb3 in term 1.
I20260812 02:32:52.228052 28239 leader_election.cc:302] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0757c63cdc194341bb7b7c499d780cb3, 0953fdae3ce746fc8a1b8f67312df383, d418d316cfa94ceaabe354287bfe46ed; no voters:
I20260812 02:32:52.228459 28792 raft_consensus.cc:2781] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:52.228969 28792 raft_consensus.cc:686] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 0757c63cdc194341bb7b7c499d780cb3, State: Running, Role: LEADER
I20260812 02:32:52.229588 28792 consensus_queue.cc:237] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } }
W20260812 02:32:52.230276 28749 tablet.cc:2369] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:52.234659 28119 catalog_manager.cc:5495] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0757c63cdc194341bb7b7c499d780cb3 (127.12.8.193). New cstate: current_term: 1 leader_uuid: "0757c63cdc194341bb7b7c499d780cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0757c63cdc194341bb7b7c499d780cb3" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 } health_report { overall_health: UNKNOWN } } }
I20260812 02:32:52.286625 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.281399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.12.8.193:46339)
0812 02:32:52.281581 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.281585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.281604 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:32:52.281702 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.281706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.281814 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.281959 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:52.282286 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.283068 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.284468 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.284880 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.284972 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.285522 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:52.286090 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:52.286239 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.286337 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.286408 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:32:52.295281 28692 raft_consensus.cc:1261] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 1 FOLLOWER]: Refusing update from remote peer 0757c63cdc194341bb7b7c499d780cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:52.295290 28529 raft_consensus.cc:1261] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 1 FOLLOWER]: Refusing update from remote peer 0757c63cdc194341bb7b7c499d780cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:52.296355 28792 consensus_queue.cc:1035] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d418d316cfa94ceaabe354287bfe46ed" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:52.296693 28796 consensus_queue.cc:1035] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0953fdae3ce746fc8a1b8f67312df383" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 34381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:52.321915 28118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:52.322252 28118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:52.326553 28118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35af554318784f17a72635501ca5e8fb: Sending DeleteTablet for 3 replicas of tablet f0124fb3ee3442d0b7710b674d3ab054
I20260812 02:32:52.327999 28497 tablet_service.cc:1514] Processing DeleteTablet for tablet f0124fb3ee3442d0b7710b674d3ab054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:52 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:53008
I20260812 02:32:52.328038 28302 tablet_service.cc:1514] Processing DeleteTablet for tablet f0124fb3ee3442d0b7710b674d3ab054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:52 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:56456
I20260812 02:32:52.328248 28662 tablet_service.cc:1514] Processing DeleteTablet for tablet f0124fb3ee3442d0b7710b674d3ab054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:52 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33880
I20260812 02:32:52.334558 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 28167
I20260812 02:32:52.349567 28811 tablet_replica.cc:330] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: stopping tablet replica
I20260812 02:32:52.349363 28810 tablet_replica.cc:330] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3: stopping tablet replica
I20260812 02:32:52.350128 28810 raft_consensus.cc:2229] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:52.350123 28811 raft_consensus.cc:2229] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:52.350266 28812 tablet_replica.cc:330] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: stopping tablet replica
I20260812 02:32:52.350695 28811 raft_consensus.cc:2258] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:52.350705 28812 raft_consensus.cc:2229] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:52.351207 28812 raft_consensus.cc:2258] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:52.352160 28810 raft_consensus.cc:2258] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:52.353116 28811 ts_tablet_manager.cc:1905] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:52.353626 28812 ts_tablet_manager.cc:1905] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:52.356590 28810 ts_tablet_manager.cc:1905] T f0124fb3ee3442d0b7710b674d3ab054 P 0757c63cdc194341bb7b7c499d780cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:52.361166 28812 ts_tablet_manager.cc:1918] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:52.361378 28812 log.cc:1196] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/f0124fb3ee3442d0b7710b674d3ab054
I20260812 02:32:52.361675 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.336768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.12.8.193:46339)
0812 02:32:52.336972 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.336978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.336998 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:32:52.337066 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.337072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.337159 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.337307 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:52.340271 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.341574 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.343709 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.344461 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.344603 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.348390 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:52.348405 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:52.348426 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:52.348494 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:52.353998 (+ 5504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.356851 (+ 2853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.356874 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.356878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.357070 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.357628 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.357634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.357637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.358160 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:52.358490 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.360747 (+ 2257us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.361032 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:32:52.361984 28812 ts_tablet_manager.cc:1939] T f0124fb3ee3442d0b7710b674d3ab054 P 0953fdae3ce746fc8a1b8f67312df383: Deleting consensus metadata
I20260812 02:32:52.362624 28389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:52.364025 28811 ts_tablet_manager.cc:1918] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:52.364223 28811 log.cc:1196] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/f0124fb3ee3442d0b7710b674d3ab054
I20260812 02:32:52.364281 28103 catalog_manager.cc:4841] TS 0953fdae3ce746fc8a1b8f67312df383 (127.12.8.195:34381): tablet f0124fb3ee3442d0b7710b674d3ab054 (table test-table [id=0ebd625c7f0a4d70b3ebceb2e64060d9]) successfully deleted
I20260812 02:32:52.364780 28811 ts_tablet_manager.cc:1939] T f0124fb3ee3442d0b7710b674d3ab054 P d418d316cfa94ceaabe354287bfe46ed: Deleting consensus metadata
I20260812 02:32:52.366524 28101 catalog_manager.cc:4841] TS d418d316cfa94ceaabe354287bfe46ed (127.12.8.194:39609): tablet f0124fb3ee3442d0b7710b674d3ab054 (table test-table [id=0ebd625c7f0a4d70b3ebceb2e64060d9]) successfully deleted
I20260812 02:32:52.464366 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 28167
I20260812 02:32:52.485141 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 28402
W20260812 02:32:52.485206 28103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.8.193:46339 (error 108)
I20260812 02:32:52.487139 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.486740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46339 (local address 127.0.0.1:56484)
0812 02:32:52.486895 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.486997 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 02:32:52.487375 28103 catalog_manager.cc:4584] TS 0757c63cdc194341bb7b7c499d780cb3 (127.12.8.193:46339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0124fb3ee3442d0b7710b674d3ab054: Network error: Client connection negotiation failed: client connection to 127.12.8.193:46339: connect: Connection refused (error 111)
I20260812 02:32:52.499501 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.486264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.12.8.194:39609)
0812 02:32:52.486434 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.486439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.486455 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:32:52.486517 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.486522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.486600 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.486694 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:52.488267 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.489054 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.490450 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.490865 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.490969 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.493630 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:52.493645 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:52.493661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:52.493703 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:52.496634 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.497451 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.497459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.497461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.497604 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.497977 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.497980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.497982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.498365 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:52.498603 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.499017 (+ 414us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.499200 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:32:52.500180 28575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:52.540927 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.540429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46339 (local address 127.0.0.1:56498)
0812 02:32:52.540645 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.540755 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 02:32:52.542045 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.541673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46339 (local address 127.0.0.1:56502)
0812 02:32:52.541821 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.541920 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 02:32:52.602648 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.602180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46339 (local address 127.0.0.1:56510)
0812 02:32:52.602388 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.602478 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 02:32:52.603915 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.603477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46339 (local address 127.0.0.1:56512)
0812 02:32:52.603662 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.603754 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 02:32:52.607108 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 28402
I20260812 02:32:52.627575 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 28586
I20260812 02:32:52.642063 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.628481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.12.8.195:34381)
0812 02:32:52.628666 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:52.628671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:52.628692 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:32:52.628784 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:52.628788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:52.628859 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:52.628946 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:52.630359 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.631148 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.632443 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:52.632862 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:52.632965 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:52.635759 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:52.635778 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:52.635795 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:52.635860 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:52.639048 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.639872 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.639882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.639884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.640051 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:52.640425 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:52.640429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:52.640431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:52.640894 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:52.641138 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:52.641548 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 02:32:52.641740 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:32:52.642761 28742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:52.685009 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.684443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46339 (local address 127.0.0.1:56516)
0812 02:32:52.684700 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.684812 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 02:32:52.686159 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:52.685762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46339 (local address 127.0.0.1:56524)
0812 02:32:52.685978 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:52.686051 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:46339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260812 02:32:52.732913 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 28586
I20260812 02:32:52.753700 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 28077
I20260812 02:32:52.754997 28158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:52.875618 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 28077
I20260812 02:32:52.911607 12323 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 02:32:53.259390 12323 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 02:32:53.259719 12323 mini_ranger.cc:66] Stopping Ranger...
I20260812 02:32:53.616933 12323 mini_ranger.cc:68] Stopped Ranger
2026-08-12 02:32:53.617 UTC [26891] LOG: received smart shutdown request
2026-08-12 02:32:53.621 UTC [26891] LOG: background worker "logical replication launcher" (PID 26901) exited with exit code 1
2026-08-12 02:32:53.621 UTC [26895] LOG: shutting down
2026-08-12 02:32:53.785 UTC [26891] LOG: database system is shut down
2026-08-12T02:32:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59655 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:54 dist-test-slave-vq2q krb5kdc[28830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:54 dist-test-slave-vq2q krb5kdc[28830](info): set up 2 sockets
Aug 12 02:32:54 dist-test-slave-vq2q krb5kdc[28830](info): commencing operation
krb5kdc: starting...
W20260812 02:32:55.082206 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.007s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:32:55 dist-test-slave-vq2q krb5kdc[28830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:32:55.469599 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:44457
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:44457,127.12.8.253:42329,127.12.8.252:45557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:55.916093 28849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:55.916774 28849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:55.916962 28849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:55.927557 28849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:55.927811 28849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:55.927910 28849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:55.928017 28849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:55.928097 28849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:55.940907 28849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:44457,127.12.8.253:42329,127.12.8.252:45557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:55.942943 28849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:55.944792 28849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:55.960721 28855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:55.960772 28858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:55.960675 28856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:55.962018 28849 server_base.cc:1034] running on GCE node
I20260812 02:32:55.963176 28849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:55.965447 28849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:55.966768 28849 hybrid_clock.cc:648] HybridClock initialized: now 1786501975966720 us; error 66 us; skew 500 ppm
I20260812 02:32:55.967324 28849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:55 dist-test-slave-vq2q krb5kdc[28830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501975, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:55.975050 28849 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:32:55.977007 28849 webserver.cc:460] Webserver started at http://127.12.8.254:40109/ using document root <none> and password file <none>
I20260812 02:32:55.977960 28849 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:55.978088 28849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:55.978425 28849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:55.982570 28849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "d49e5c94af87443bbd44fdc96cb2e5fa"
format_stamp: "Formatted at 2026-08-12 02:32:55 on dist-test-slave-vq2q"
server_key: "a2d20f27ad24a592be786d22934cfb50"
server_key_iv: "1dc69ccb27d8158afdcda7c320f96c59"
server_key_version: "encryptionkey@0"
I20260812 02:32:55.983707 28849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "d49e5c94af87443bbd44fdc96cb2e5fa"
format_stamp: "Formatted at 2026-08-12 02:32:55 on dist-test-slave-vq2q"
server_key: "a2d20f27ad24a592be786d22934cfb50"
server_key_iv: "1dc69ccb27d8158afdcda7c320f96c59"
server_key_version: "encryptionkey@0"
I20260812 02:32:55.991047 28849 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 02:32:55.995687 28871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:55.997651 28849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 02:32:55.997879 28849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "d49e5c94af87443bbd44fdc96cb2e5fa"
format_stamp: "Formatted at 2026-08-12 02:32:55 on dist-test-slave-vq2q"
server_key: "a2d20f27ad24a592be786d22934cfb50"
server_key_iv: "1dc69ccb27d8158afdcda7c320f96c59"
server_key_version: "encryptionkey@0"
I20260812 02:32:55.998178 28849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:56.024055 28849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:56.030112 28849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:56.030476 28849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:56.051858 28849 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:44457
I20260812 02:32:56.051896 28930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:44457 every 8 connection(s)
I20260812 02:32:56.053690 28849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:56.059098 28932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:56.063134 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 28849
I20260812 02:32:56.063405 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:32:56.064162 12323 external_mini_cluster.cc:1428] Setting key 88f8250d870e8fb894524708b966d17a
I20260812 02:32:56.068153 28932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:56 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501975, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:56.097748 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.072446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.12.8.254:44457)
0812 02:32:56.073686 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:56.073703 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:56.074385 (+ 682us) server_negotiation.cc:407] Connection header received
0812 02:32:56.076950 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:56.076958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:56.077403 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:56.077684 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:56.079218 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.080700 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.082988 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.083717 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.083908 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:56.088825 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:56.088835 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:56.088853 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:56.088900 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:56.092265 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:56.093248 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:56.093256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:56.093259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:56.093476 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:56.093930 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:56.093934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:56.093936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:56.094653 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:56.094893 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:56.095772 (+ 879us) server_negotiation.cc:299] Negotiation successful
0812 02:32:56.096204 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":620,"threads_started":1}
I20260812 02:32:56.097806 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.072446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44457 (local address 127.0.0.1:37932)
0812 02:32:56.074147 (+ 1701us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.074217 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:56.076432 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:56.077739 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:56.077746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:56.078521 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:56.078975 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:56.078996 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.080892 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.080899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:56.082767 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:56.082789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.084052 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.084064 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:56.084251 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:56.085131 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:56.085164 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:56.088501 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:56.092495 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:56.092510 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:56.092525 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:56.093042 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:56.093595 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:56.093600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:56.093604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:56.093812 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:56.094878 (+ 1066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:56.094889 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:56.095226 (+ 337us) client_negotiation.cc:769] Sending connection context
0812 02:32:56.095624 (+ 398us) client_negotiation.cc:240] Negotiation successful
0812 02:32:56.096411 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"mutex_wait_us":122,"spinlock_wait_cycles":87808,"thread_start_us":1073,"threads_started":1}
WARNING: no policy specified for kudu/127.12.8.253@KRBTEST.COM; defaulting to no policy
I20260812 02:32:56.101814 28932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.253@KRBTEST.COM" created.
I20260812 02:32:56.103417 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.102785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55096)
0812 02:32:56.103047 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.103223 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260812 02:32:56.103798 28876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
I20260812 02:32:56.105871 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.105392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55100)
0812 02:32:56.105597 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.105684 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 02:32:56.106096 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:42329: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab.
Entry for principal kudu/127.12.8.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab.
I20260812 02:32:56.145673 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.253:42329
--webserver_interface=127.12.8.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab
--principal=kudu/127.12.8.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:44457,127.12.8.253:42329,127.12.8.252:45557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:32:56.155539 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } attempt: 1
I20260812 02:32:56.157012 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.156456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55106)
0812 02:32:56.156703 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.156829 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 02:32:56.158401 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.157914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55120)
0812 02:32:56.158087 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.158196 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 02:32:56.158681 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:42329: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
I20260812 02:32:56.227005 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } attempt: 2
I20260812 02:32:56.228538 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.227932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55128)
0812 02:32:56.228237 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.228362 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 02:32:56.229902 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.229468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55142)
0812 02:32:56.229644 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.229730 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 02:32:56.230195 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:42329: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
I20260812 02:32:56.321462 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } attempt: 3
I20260812 02:32:56.323088 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.322472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55144)
0812 02:32:56.322753 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.322888 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260812 02:32:56.324368 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.323889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55150)
0812 02:32:56.324075 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.324185 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260812 02:32:56.324606 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:42329: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
I20260812 02:32:56.467906 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } attempt: 4
I20260812 02:32:56.469743 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.468948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55160)
0812 02:32:56.469430 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.469563 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260812 02:32:56.471114 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.470674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55174)
0812 02:32:56.470850 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.470939 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 02:32:56.471350 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:42329: Network error: Client connection negotiation failed: client connection to 127.12.8.253:42329: connect: Connection refused (error 111)
W20260812 02:32:56.582155 28941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:56.582578 28941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:56.582731 28941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:56.591171 28941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:56.591351 28941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:56.591416 28941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:56.591509 28941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:56.591571 28941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:56.603567 28941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:44457,127.12.8.253:42329,127.12.8.252:45557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab
--principal=kudu/127.12.8.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.253:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.8.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:56.605587 28941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:56.607306 28941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:56.623311 28946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:56.623215 28950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:56.623260 28948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:56.624405 28941 server_base.cc:1034] running on GCE node
I20260812 02:32:56.625612 28941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:56.627676 28941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:56.628952 28941 hybrid_clock.cc:648] HybridClock initialized: now 1786501976628896 us; error 57 us; skew 500 ppm
I20260812 02:32:56.629498 28941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:56 dist-test-slave-vq2q krb5kdc[28830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501976, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:56.633512 28941 init.cc:377] Logged in from keytab as kudu/127.12.8.253@KRBTEST.COM (short username kudu)
I20260812 02:32:56.635651 28941 webserver.cc:460] Webserver started at http://127.12.8.253:37627/ using document root <none> and password file <none>
I20260812 02:32:56.636629 28941 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:56.636786 28941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:56.637252 28941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:56.641363 28941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/data/instance:
uuid: "040f95efa45247b1a4a8b6cfa0f27305"
format_stamp: "Formatted at 2026-08-12 02:32:56 on dist-test-slave-vq2q"
server_key: "386314627c13e70ef9a0ac0191700409"
server_key_iv: "a966737b001f7f5f24d13a6d45ab1054"
server_key_version: "encryptionkey@0"
I20260812 02:32:56.642508 28941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/wal/instance:
uuid: "040f95efa45247b1a4a8b6cfa0f27305"
format_stamp: "Formatted at 2026-08-12 02:32:56 on dist-test-slave-vq2q"
server_key: "386314627c13e70ef9a0ac0191700409"
server_key_iv: "a966737b001f7f5f24d13a6d45ab1054"
server_key_version: "encryptionkey@0"
I20260812 02:32:56.649961 28941 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 02:32:56.654578 28958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:56.656306 28941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:32:56.656548 28941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/wal
uuid: "040f95efa45247b1a4a8b6cfa0f27305"
format_stamp: "Formatted at 2026-08-12 02:32:56 on dist-test-slave-vq2q"
server_key: "386314627c13e70ef9a0ac0191700409"
server_key_iv: "a966737b001f7f5f24d13a6d45ab1054"
server_key_version: "encryptionkey@0"
I20260812 02:32:56.656797 28941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:56.679131 28941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:56.684336 28941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:56.684702 28941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:56.706861 28941 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.253:42329
I20260812 02:32:56.706883 29026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.253:42329 every 8 connection(s)
I20260812 02:32:56.708599 28941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
I20260812 02:32:56.713735 29028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:56.717970 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 28941
I20260812 02:32:56.718184 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-1/wal/instance
I20260812 02:32:56.718724 12323 external_mini_cluster.cc:1428] Setting key 12493e485639cd24d38a862bbb5a2e23
I20260812 02:32:56.722500 29028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:56 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501976, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:56.749430 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.726189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.12.8.254:44457)
0812 02:32:56.726861 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:56.726866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:56.728501 (+ 1635us) server_negotiation.cc:407] Connection header received
0812 02:32:56.730985 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:56.730992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:56.731113 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:56.731227 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:56.733354 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.734206 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.736622 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.737104 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.737262 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:56.741758 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:56.741764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:56.741768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:56.741817 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:56.744805 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:56.746052 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:56.746057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:56.746060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:56.746269 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:56.746956 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:56.746961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:56.746963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:56.747346 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:56.747501 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:56.748747 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0812 02:32:56.749010 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":390,"threads_started":1}
I20260812 02:32:56.751236 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.727148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44457 (local address 127.0.0.1:37938)
0812 02:32:56.728249 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.728311 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:56.730581 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:56.731584 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:56.731595 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:56.732294 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:56.733078 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:56.733100 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.734403 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.734410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:56.736427 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:56.736445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.737389 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.737398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:56.737681 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:56.738537 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:56.738564 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:56.741433 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:56.745020 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:56.745033 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:56.745059 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:56.745864 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:56.746401 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:56.746406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:56.746409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:56.746774 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:56.747500 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:56.747512 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:56.748164 (+ 652us) client_negotiation.cc:769] Sending connection context
0812 02:32:56.748584 (+ 420us) client_negotiation.cc:240] Negotiation successful
0812 02:32:56.749351 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":490,"threads_started":1}
I20260812 02:32:56.753561 29028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.8.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.252@KRBTEST.COM" created.
Aug 12 02:32:56 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501976, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:32:56.770599 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } attempt: 5
I20260812 02:32:56.771286 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.754701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55180)
0812 02:32:56.754944 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.754966 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:56.755103 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:56.756371 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:56.756375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:56.756808 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:56.756993 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:56.757004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.758617 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.758623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:56.759559 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:56.759573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.760560 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.760567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:56.760787 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:56.761574 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:56.761598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:56.764513 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:56.768255 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:56.768261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:56.768265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:56.768829 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:56.769279 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:56.769284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:56.769288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:56.769462 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:56.770178 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:56.770184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:56.770354 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 02:32:56.770484 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 02:32:56.770848 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 02:32:56.771370 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.754956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.12.8.253:42329)
0812 02:32:56.755810 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:56.755820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:56.755860 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:32:56.755927 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:56.755936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:56.756202 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:56.756346 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:56.757203 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.758473 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.759721 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.760374 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.760501 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:56.764818 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:56.764833 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:56.764852 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:56.764894 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:56.768068 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:56.768984 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:56.768992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:56.768995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:56.769145 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:56.769573 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:56.769578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:56.769581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:56.770030 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:56.770260 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:56.770665 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 02:32:56.770980 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":639,"thread_start_us":458,"threads_started":1}
I20260812 02:32:56.775454 29028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:32:56.777020 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.776431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48476)
0812 02:32:56.776670 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.776806 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 02:32:56.777360 28959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
I20260812 02:32:56.779680 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.779205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48490)
0812 02:32:56.779380 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.779489 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 02:32:56.780092 29028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Aug 12 02:32:56 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501975, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:32:56.788584 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.772018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.12.8.253:42329)
0812 02:32:56.772786 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:56.772793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:56.772817 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:32:56.772865 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:56.772869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:56.772945 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:56.773054 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:56.774173 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.775303 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.777257 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.777937 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.778766 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:56.782228 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:56.782235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:56.782238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:56.782276 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:56.785250 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:56.786368 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:56.786374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:56.786376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:56.786546 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:56.786987 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:56.786991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:56.786993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:56.787362 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:56.787491 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:56.787973 (+ 482us) server_negotiation.cc:299] Negotiation successful
0812 02:32:56.788200 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":378,"threads_started":1}
I20260812 02:32:56.788666 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.771417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55182)
0812 02:32:56.771666 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.771691 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:56.771824 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:56.773151 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:56.773157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:56.773733 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:56.773997 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:56.774014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.775483 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.775489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:56.777031 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:56.777052 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:56.778144 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:56.778152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:56.778371 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:56.779124 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:56.779152 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:56.781947 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:56.785465 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:56.785472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:56.785476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:56.786202 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:56.786671 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:56.786676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:56.786679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:56.786867 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:56.787488 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:56.787493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:56.787713 (+ 220us) client_negotiation.cc:769] Sending connection context
0812 02:32:56.787874 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 02:32:56.788312 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 02:32:56.789924 28932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:32:56.791420 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.790837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48502)
0812 02:32:56.791065 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.791183 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 02:32:56.792531 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.792090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48510)
0812 02:32:56.792261 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.792377 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 02:32:56.792748 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab.
Entry for principal kudu/127.12.8.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab.
I20260812 02:32:56.805609 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.252:45557
--webserver_interface=127.12.8.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab
--principal=kudu/127.12.8.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:44457,127.12.8.253:42329,127.12.8.252:45557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:32:56.829571 29028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 1
I20260812 02:32:56.831219 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.830590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48522)
0812 02:32:56.830852 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.830995 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 02:32:56.832471 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.832046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48530)
0812 02:32:56.832205 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.832317 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 02:32:56.832751 29028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
I20260812 02:32:56.844034 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 1
I20260812 02:32:56.845583 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.844963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48536)
0812 02:32:56.845247 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.845377 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 02:32:56.846743 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.846337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48546)
0812 02:32:56.846521 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.846616 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 02:32:56.847038 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
I20260812 02:32:56.901013 29028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 2
I20260812 02:32:56.902554 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.901997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48562)
0812 02:32:56.902242 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.902352 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 02:32:56.905653 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.904975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48564)
0812 02:32:56.905298 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.905411 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260812 02:32:56.905975 29028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
I20260812 02:32:56.915313 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 2
I20260812 02:32:56.917605 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.916546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48572)
0812 02:32:56.917055 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.917276 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
I20260812 02:32:56.920497 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.918707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48576)
0812 02:32:56.919688 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.919949 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":382,"threads_started":1}
W20260812 02:32:56.920950 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
I20260812 02:32:56.997351 29028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 3
I20260812 02:32:56.998878 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:56.998253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48580)
0812 02:32:56.998528 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:56.998646 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 02:32:57.000784 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.000091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48590)
0812 02:32:57.000477 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.000592 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
W20260812 02:32:57.001235 29028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
I20260812 02:32:57.027369 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 3
I20260812 02:32:57.028980 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.028395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48602)
0812 02:32:57.028656 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.028779 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 02:32:57.030336 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.029836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48608)
0812 02:32:57.030078 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.030181 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260812 02:32:57.030691 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
I20260812 02:32:57.144555 29028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 4
I20260812 02:32:57.146339 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.145602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48610)
0812 02:32:57.145946 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.146114 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260812 02:32:57.147711 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.147185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48624)
0812 02:32:57.147391 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.147530 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 02:32:57.147940 29028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
I20260812 02:32:57.207926 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 4
I20260812 02:32:57.209470 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.208805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48640)
0812 02:32:57.209107 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.209281 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260812 02:32:57.210646 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.210221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48652)
0812 02:32:57.210391 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.210474 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 02:32:57.210901 28932 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:45557: Network error: Client connection negotiation failed: client connection to 127.12.8.252:45557: connect: Connection refused (error 111)
W20260812 02:32:57.215565 29039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:57.215896 29039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:57.216037 29039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:57.223543 29039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:57.223685 29039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:57.223768 29039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:57.223861 29039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:57.223932 29039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:57.234457 29039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:44457,127.12.8.253:42329,127.12.8.252:45557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab
--principal=kudu/127.12.8.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.252:45557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.8.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:32:57.235992 29039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:57.237462 29039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:57.251614 29046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:57.251583 29049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:57.251648 29047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:57.252789 29039 server_base.cc:1034] running on GCE node
I20260812 02:32:57.253787 29039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:57.255419 29039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:57.256685 29039 hybrid_clock.cc:648] HybridClock initialized: now 1786501977256622 us; error 60 us; skew 500 ppm
I20260812 02:32:57.257109 29039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501977, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:57.260635 29039 init.cc:377] Logged in from keytab as kudu/127.12.8.252@KRBTEST.COM (short username kudu)
I20260812 02:32:57.262391 29039 webserver.cc:460] Webserver started at http://127.12.8.252:42473/ using document root <none> and password file <none>
I20260812 02:32:57.263128 29039 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:57.263238 29039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:57.263579 29039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:57.266836 29039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/data/instance:
uuid: "a6425f1e3dce4870907c5ce090240fc8"
format_stamp: "Formatted at 2026-08-12 02:32:57 on dist-test-slave-vq2q"
server_key: "f29f68e96604e3512b33bf2ff807c055"
server_key_iv: "5d2bf963a90e03cafc2ee3331356a0b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:57.267683 29039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/wal/instance:
uuid: "a6425f1e3dce4870907c5ce090240fc8"
format_stamp: "Formatted at 2026-08-12 02:32:57 on dist-test-slave-vq2q"
server_key: "f29f68e96604e3512b33bf2ff807c055"
server_key_iv: "5d2bf963a90e03cafc2ee3331356a0b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:57.273730 29039 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 02:32:57.277976 29057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:57.279693 29039 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:32:57.279923 29039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/wal
uuid: "a6425f1e3dce4870907c5ce090240fc8"
format_stamp: "Formatted at 2026-08-12 02:32:57 on dist-test-slave-vq2q"
server_key: "f29f68e96604e3512b33bf2ff807c055"
server_key_iv: "5d2bf963a90e03cafc2ee3331356a0b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:57.280136 29039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:57.308677 29039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:57.313216 29039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:57.313511 29039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:57.334708 29039 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.252:45557
I20260812 02:32:57.334796 29114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.252:45557 every 8 connection(s)
I20260812 02:32:57.336261 29039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
I20260812 02:32:57.340209 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 29039
I20260812 02:32:57.340443 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501741404412-12323-0/minicluster-data/master-2/wal/instance
I20260812 02:32:57.340516 29115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:57.341039 12323 external_mini_cluster.cc:1428] Setting key d8b542c34c2ec97b01199505d22dea7f
I20260812 02:32:57.351547 29115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501977, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:57.370946 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.343541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.12.8.252:45557)
0812 02:32:57.344656 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.344675 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.344742 (+ 67us) server_negotiation.cc:407] Connection header received
0812 02:32:57.347695 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.347733 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.348374 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.348932 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.350864 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.352993 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.354868 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.356074 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.356332 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.361414 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.361424 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.361439 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.361501 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.365245 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.366092 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.366099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.366107 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.366295 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.366716 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.366720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.366723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.367446 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.367729 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.368478 (+ 749us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.369128 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":462,"threads_started":1}
I20260812 02:32:57.373499 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.355305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.12.8.254:44457)
0812 02:32:57.356012 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.356019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.356806 (+ 787us) server_negotiation.cc:407] Connection header received
0812 02:32:57.356884 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.356888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.356983 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.357105 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.358767 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.359540 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.361166 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.361754 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.362004 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.366172 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.366178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.366181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.366245 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.369682 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.370561 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.370568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.370572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.370771 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.371554 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.371559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.371563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.372009 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.372194 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.372858 (+ 664us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.373087 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":307,"threads_started":1}
I20260812 02:32:57.373608 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.355304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44457 (local address 127.0.0.1:37946)
0812 02:32:57.356472 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.356529 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:57.356691 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:57.357331 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:57.357344 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:57.358175 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:57.358548 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.358559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.359714 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.359719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.360997 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.361014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.361968 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.361975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.362198 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.362915 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:57.362951 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:57.365896 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:57.369896 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.369908 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.369913 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.370416 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.370947 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.370951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.370954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.371151 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.372262 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:57.372270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:57.372494 (+ 224us) client_negotiation.cc:769] Sending connection context
0812 02:32:57.372697 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 02:32:57.373072 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":633,"threads_started":1}
I20260812 02:32:57.375739 29115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501977, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:32:57.393931 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.376996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:42329 (local address 127.0.0.1:55184)
0812 02:32:57.377320 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.377338 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:57.377447 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:57.377868 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:57.377873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:57.378327 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:57.378483 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.378492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.379887 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.379893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.381298 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.381316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.382379 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.382385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.382599 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.383340 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:57.383363 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:57.386260 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:57.390392 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.390399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.390402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.390989 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.391489 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.391493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.391497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.391667 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.392720 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:57.392725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:57.392930 (+ 205us) client_negotiation.cc:769] Sending connection context
0812 02:32:57.393092 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 02:32:57.393534 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260812 02:32:57.393934 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.376928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.12.8.253:42329)
0812 02:32:57.377516 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.377521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.377539 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:32:57.377617 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.377621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.377697 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.377802 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.378674 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.379717 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.381501 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.382190 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.382352 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.386544 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.386553 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.386557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.386605 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.390163 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.391165 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.391172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.391175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.391344 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.391790 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.391795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.391799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.392187 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.392352 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.393284 (+ 932us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.393510 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":355,"threads_started":1}
I20260812 02:32:57.394673 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.375444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.12.8.254:44457)
0812 02:32:57.375659 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.375665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.375810 (+ 145us) server_negotiation.cc:407] Connection header received
0812 02:32:57.376191 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.376198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.376319 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.376443 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.378284 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.379478 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.381483 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.382199 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.382340 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.387752 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.387760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.387765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.387821 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.391123 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.392033 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.392038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.392041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.392192 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.392679 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.392682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.392684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.393001 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.393315 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.393747 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.394007 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:32:57.395740 29115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501977, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:32:57.410619 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.396865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.12.8.252:45557)
0812 02:32:57.397038 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.397044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.397060 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:32:57.397100 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.397104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.397156 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.397297 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.398007 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.398700 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.399847 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.400253 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.400357 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.404362 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.404367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.404370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.404402 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.407528 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.408405 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.408411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.408415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.408556 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.408917 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.408920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.408922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.409326 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.409472 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.409856 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.410289 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":66}
I20260812 02:32:57.410626 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.396619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48666)
0812 02:32:57.396849 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.396871 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:57.396980 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:57.397310 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:57.397315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:57.397718 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:57.397878 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.397886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.398820 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.398825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.399714 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.399725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.400402 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.400408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.400587 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.401325 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:57.401344 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:57.404070 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:57.407739 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.407745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.407749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.408255 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.408668 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.408673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.408677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.408813 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.409434 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:57.409439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:57.409606 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 02:32:57.409741 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 02:32:57.410122 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260812 02:32:57.413705 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.397255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.12.8.253:42329)
0812 02:32:57.397454 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.397460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.397480 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:32:57.397619 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.397625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.397698 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.397785 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.399245 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.400035 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.401602 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.402053 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.402180 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.407869 (+ 5689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.407876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.407879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.407945 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.410594 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.411512 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.411516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.411519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.411685 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.412191 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.412196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.412199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.412511 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.412733 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.413240 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.413457 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:32:57.418154 29115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8: Bootstrap starting.
I20260812 02:32:57.420562 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:32:57.423573 29115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:57.425354 29115 log.cc:824] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:57.430228 29115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8: No bootstrap required, opened a new log
I20260812 02:32:57.444501 29115 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.445459 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.427774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.12.8.254:44457)
0812 02:32:57.427949 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.427955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.429720 (+ 1765us) server_negotiation.cc:407] Connection header received
0812 02:32:57.429898 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.429904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.429979 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.430063 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.431816 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.432563 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.434232 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.434869 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.434965 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.438481 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.438488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.438492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.438535 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.442017 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.442998 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.443003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.443007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.443157 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.443655 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.443660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.443664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.444035 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.444279 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.444831 (+ 552us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.445056 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:32:57.445451 29115 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:57.445861 29115 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6425f1e3dce4870907c5ce090240fc8, State: Initialized, Role: FOLLOWER
I20260812 02:32:57.446966 29115 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.447198 29028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 5
I20260812 02:32:57.452498 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.426625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.12.8.252:45557)
0812 02:32:57.426908 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.426917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.433473 (+ 6556us) server_negotiation.cc:407] Connection header received
0812 02:32:57.433760 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.433766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.433864 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.433941 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.436573 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.437808 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.440064 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.440919 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.441024 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.444591 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.444598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.444606 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.444651 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.448274 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.449098 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.449104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.449108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.449310 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.449775 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.449780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.449784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.450219 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.450524 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.451280 (+ 756us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.451932 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 02:32:57.454870 29136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.455440 29136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:57.457926 29115 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:57.459882 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.425793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.12.8.253:42329)
0812 02:32:57.425983 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.425990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.428857 (+ 2867us) server_negotiation.cc:407] Connection header received
0812 02:32:57.429091 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.429099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.429271 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.429422 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.432597 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.433808 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.436575 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.437341 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.437458 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.441987 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.441996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.442001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.442057 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.447573 (+ 5516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.456339 (+ 8766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.456348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.456353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.456644 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.457374 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.457381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.457384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.457914 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.458267 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.459019 (+ 752us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.459383 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501976, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
I20260812 02:32:57.471793 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.447884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48682)
0812 02:32:57.448675 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.448693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:57.448805 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:57.453832 (+ 5027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:57.453839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:57.454444 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:57.454700 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.454713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.456436 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.456443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.458097 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.458116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.459273 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.459280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.459493 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.460230 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:57.460267 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:57.463773 (+ 3506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:57.468466 (+ 4693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.468473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.468479 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.469286 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.469707 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.469713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.469718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.469931 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.470638 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:57.470645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:57.470853 (+ 208us) client_negotiation.cc:769] Sending connection context
0812 02:32:57.470987 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 02:32:57.471352 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654}
I20260812 02:32:57.471809 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.448438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.12.8.252:45557)
0812 02:32:57.453464 (+ 5026us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.453471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.453493 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:32:57.453548 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.453554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.453638 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.454863 (+ 1225us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.455005 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.456273 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.458346 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.459078 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.461285 (+ 2207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.464011 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.464016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.464019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.464049 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.468069 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.469428 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.469432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.469435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.469572 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.470074 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.470078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.470080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.470485 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.470653 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.471125 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.471372 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4891,"thread_start_us":12027,"threads_started":1}
I20260812 02:32:57.477300 29028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305: Bootstrap starting.
I20260812 02:32:57.479321 29152 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:32:57.481202 29028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:57.482429 29028 log.cc:824] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:57.486019 29028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305: No bootstrap required, opened a new log
I20260812 02:32:57.488137 28932 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } attempt: 5
I20260812 02:32:57.482568 29039 master_runner.cc:186] Error getting master registration for 127.12.8.254:44457: OK, instance_id { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" instance_seqno: 1786501976045712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:32:57.494012 29039 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:32:57.496729 29028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.497085 29028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:57.497251 29028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040f95efa45247b1a4a8b6cfa0f27305, State: Initialized, Role: FOLLOWER
Aug 12 02:32:57 dist-test-slave-vq2q krb5kdc[28830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501975, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
I20260812 02:32:57.497980 29028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.499156 29155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.499727 29155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:57.500279 29028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:57.505748 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.488811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:45557 (local address 127.0.0.1:48692)
0812 02:32:57.489110 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:57.489129 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:57.489311 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:57.489650 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:57.489655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:57.490163 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:57.490364 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.490374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.491559 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.491563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.492602 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:57.492616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.494575 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.494581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:57.494837 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.495626 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:57.495652 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:57.498735 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:57.502631 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.502637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.502642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.503263 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.503637 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:57.503643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:57.503647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:57.503828 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:57.504510 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:57.504515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:57.504747 (+ 232us) client_negotiation.cc:769] Sending connection context
0812 02:32:57.504908 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 02:32:57.505286 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260812 02:32:57.506829 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:57.488881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.12.8.252:45557)
0812 02:32:57.489092 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:57.489099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:57.489242 (+ 143us) server_negotiation.cc:407] Connection header received
0812 02:32:57.489438 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:57.489444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:57.489519 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:57.489605 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:57.490522 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.491384 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.492737 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:57.493332 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:57.494323 (+ 991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:57.499011 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:57.499018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:57.499023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:57.499129 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:57.502434 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.503378 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.503384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.503389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.503521 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:57.503927 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:57.503932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:57.503936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:57.504388 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:57.504535 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:57.506104 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0812 02:32:57.506379 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:32:57.513681 28932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa: Bootstrap starting.
W20260812 02:32:57.516279 29168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:57.516536 29168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:57.519341 28932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:57.509620 28941 master_runner.cc:186] Error getting master registration for 127.12.8.254:44457: OK, instance_id { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" instance_seqno: 1786501976045712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:32:57.519701 28941 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:32:57.521153 28932 log.cc:824] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:57.526038 28932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa: No bootstrap required, opened a new log
I20260812 02:32:57.538897 28932 raft_consensus.cc:348] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.539500 28932 raft_consensus.cc:374] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:57.539652 28932 raft_consensus.cc:729] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49e5c94af87443bbd44fdc96cb2e5fa, State: Initialized, Role: FOLLOWER
I20260812 02:32:57.540555 28932 consensus_queue.cc:260] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.541877 29171 sys_catalog.cc:455] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.542309 29171 sys_catalog.cc:458] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:57.543787 28932 sys_catalog.cc:564] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:32:57.556756 29182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:57.556927 29182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 02:32:57.566118 29064 tablet.cc:2369] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:57.567098 28849 master_runner.cc:263] No leader master found from master d49e5c94af87443bbd44fdc96cb2e5fa
I20260812 02:32:57.567318 28849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:32:57.618985 29155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:57.619463 29155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.621688 29155 leader_election.cc:288] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d49e5c94af87443bbd44fdc96cb2e5fa (127.12.8.254:44457), a6425f1e3dce4870907c5ce090240fc8 (127.12.8.252:45557)
I20260812 02:32:57.622713 29086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040f95efa45247b1a4a8b6cfa0f27305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6425f1e3dce4870907c5ce090240fc8" is_pre_election: true
I20260812 02:32:57.622951 28902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040f95efa45247b1a4a8b6cfa0f27305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" is_pre_election: true
I20260812 02:32:57.623474 29086 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040f95efa45247b1a4a8b6cfa0f27305 in term 0.
I20260812 02:32:57.623736 28902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040f95efa45247b1a4a8b6cfa0f27305 in term 0.
I20260812 02:32:57.624141 28959 leader_election.cc:302] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040f95efa45247b1a4a8b6cfa0f27305, a6425f1e3dce4870907c5ce090240fc8; no voters:
I20260812 02:32:57.624821 29155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:57.625002 29155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:57.625113 29155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:57.628842 29155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.630364 29086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040f95efa45247b1a4a8b6cfa0f27305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6425f1e3dce4870907c5ce090240fc8"
I20260812 02:32:57.630607 29086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:57.630566 28902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040f95efa45247b1a4a8b6cfa0f27305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa"
I20260812 02:32:57.630909 28902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:57.635092 29086 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040f95efa45247b1a4a8b6cfa0f27305 in term 1.
I20260812 02:32:57.635136 28902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040f95efa45247b1a4a8b6cfa0f27305 in term 1.
I20260812 02:32:57.635687 28959 leader_election.cc:302] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040f95efa45247b1a4a8b6cfa0f27305, a6425f1e3dce4870907c5ce090240fc8; no voters:
I20260812 02:32:57.636595 29155 leader_election.cc:288] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [CANDIDATE]: Term 1 election: Requested vote from peers d49e5c94af87443bbd44fdc96cb2e5fa (127.12.8.254:44457), a6425f1e3dce4870907c5ce090240fc8 (127.12.8.252:45557)
I20260812 02:32:57.636739 29186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:57.637120 29186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [term 1 LEADER]: Becoming Leader. State: Replica: 040f95efa45247b1a4a8b6cfa0f27305, State: Running, Role: LEADER
I20260812 02:32:57.638007 29186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } }
I20260812 02:32:57.645450 29155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040f95efa45247b1a4a8b6cfa0f27305. Latest consensus state: current_term: 1 leader_uuid: "040f95efa45247b1a4a8b6cfa0f27305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.645876 29155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:57.647707 29188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:57.650413 29188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:57.661587 29086 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [term 1 FOLLOWER]: Refusing update from remote peer 040f95efa45247b1a4a8b6cfa0f27305: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:57.665356 28902 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [term 1 FOLLOWER]: Refusing update from remote peer 040f95efa45247b1a4a8b6cfa0f27305: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:57.665417 29155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:57.666833 29155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:32:57.688599 28964 tablet.cc:2369] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:57.693549 29136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040f95efa45247b1a4a8b6cfa0f27305. Latest consensus state: current_term: 1 leader_uuid: "040f95efa45247b1a4a8b6cfa0f27305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.693871 29136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:57.699260 29189 mvcc.cc:205] Tried to move back new op lower bound from 7317512100489973760 to 7317512100412534784. Current Snapshot: MvccSnapshot[applied={T|T < 7317512100489973760}]
I20260812 02:32:57.700789 29190 mvcc.cc:205] Tried to move back new op lower bound from 7317512100489973760 to 7317512100412534784. Current Snapshot: MvccSnapshot[applied={T|T < 7317512100489973760}]
I20260812 02:32:57.705266 29136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "040f95efa45247b1a4a8b6cfa0f27305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.705541 29136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:57.707842 29171 sys_catalog.cc:455] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040f95efa45247b1a4a8b6cfa0f27305. Latest consensus state: current_term: 1 leader_uuid: "040f95efa45247b1a4a8b6cfa0f27305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.708165 29171 sys_catalog.cc:458] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:57.718051 29171 sys_catalog.cc:455] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "040f95efa45247b1a4a8b6cfa0f27305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.718437 29171 sys_catalog.cc:458] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:57.722951 29155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "040f95efa45247b1a4a8b6cfa0f27305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.723358 29155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:57.723871 29155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "040f95efa45247b1a4a8b6cfa0f27305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49e5c94af87443bbd44fdc96cb2e5fa" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 44457 } } peers { permanent_uuid: "040f95efa45247b1a4a8b6cfa0f27305" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 42329 } } peers { permanent_uuid: "a6425f1e3dce4870907c5ce090240fc8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 45557 } } }
I20260812 02:32:57.724156 29155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:57.725283 29188 catalog_manager.cc:1347] Generated new cluster ID: 462d42b8fb6a47c4ab92977efbae35a0
I20260812 02:32:57.725409 29188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:57.791451 29188 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:57.794265 29188 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:57.825376 29188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305: Generated new TSK 0
I20260812 02:32:57.827659 29188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:32:58.486415 29152 catalog_manager.cc:1259] Loaded cluster ID: 462d42b8fb6a47c4ab92977efbae35a0
I20260812 02:32:58.486580 29152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8: loading cluster ID for follower catalog manager: success
I20260812 02:32:58.489995 29152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8: acquiring CA information for follower catalog manager: success
I20260812 02:32:58.492753 29152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6425f1e3dce4870907c5ce090240fc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:32:58.559679 29182 catalog_manager.cc:1259] Loaded cluster ID: 462d42b8fb6a47c4ab92977efbae35a0
I20260812 02:32:58.559820 29182 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa: loading cluster ID for follower catalog manager: success
I20260812 02:32:58.563741 29182 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa: acquiring CA information for follower catalog manager: success
I20260812 02:32:58.566447 29182 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d49e5c94af87443bbd44fdc96cb2e5fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:32:58.942062 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 28849
I20260812 02:32:58.944013 28926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:59.070268 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 28849
W20260812 02:32:59.102066 28961 connection.cc:535] client connection to 127.12.8.254:44457 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 02:32:59.102474 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 28941
I20260812 02:32:59.103924 29022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:59.231851 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:59.103378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:44457 (local address 127.0.0.1:37988)
0812 02:32:59.103895 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:59.231457 (+127562us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:44457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":313,"threads_started":1}
W20260812 02:32:59.232239 28961 consensus_peers.cc:487] T 00000000000000000000000000000000 P 040f95efa45247b1a4a8b6cfa0f27305 -> Peer d49e5c94af87443bbd44fdc96cb2e5fa (127.12.8.254:44457): Couldn't send request to peer d49e5c94af87443bbd44fdc96cb2e5fa. Status: Network error: Client connection negotiation failed: client connection to 127.12.8.254:44457: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 02:32:59.232304 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 28941
I20260812 02:32:59.265858 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 29039
I20260812 02:32:59.267367 29110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:32:59.393056 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 29039
2026-08-12T02:32:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5503 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:59 dist-test-slave-vq2q krb5kdc[29213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:59 dist-test-slave-vq2q krb5kdc[29213](info): set up 2 sockets
Aug 12 02:32:59 dist-test-slave-vq2q krb5kdc[29213](info): commencing operation
krb5kdc: starting...
W20260812 02:33:00.587065 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.002s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:33:00 dist-test-slave-vq2q krb5kdc[29213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:33:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:33:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:33:00.955082 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:42381
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:42381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:01.389107 29228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:01.389609 29228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:01.389755 29228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:01.398140 29228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:33:01.398303 29228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:01.398357 29228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:01.398445 29228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:33:01.398522 29228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:33:01.410488 29228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:42381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:42381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:01.412379 29228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:01.414158 29228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:01.428793 29233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:01.428825 29234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:01.429021 29236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:01.430009 29228 server_base.cc:1034] running on GCE node
I20260812 02:33:01.431154 29228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:01.433153 29228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:01.434427 29228 hybrid_clock.cc:648] HybridClock initialized: now 1786501981434357 us; error 68 us; skew 500 ppm
I20260812 02:33:01.434849 29228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:01 dist-test-slave-vq2q krb5kdc[29213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501981, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:01.443490 29228 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:33:01.445748 29228 webserver.cc:460] Webserver started at http://127.12.8.254:34255/ using document root <none> and password file <none>
I20260812 02:33:01.446714 29228 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:01.446847 29228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:01.447252 29228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:01.451241 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "63cd2fa2f56a47a5994eac3ce2713632"
format_stamp: "Formatted at 2026-08-12 02:33:01 on dist-test-slave-vq2q"
server_key: "26222ca9d85856e2ea069489889d5695"
server_key_iv: "97276dd793289175a127cdb031d21426"
server_key_version: "encryptionkey@0"
I20260812 02:33:01.452280 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "63cd2fa2f56a47a5994eac3ce2713632"
format_stamp: "Formatted at 2026-08-12 02:33:01 on dist-test-slave-vq2q"
server_key: "26222ca9d85856e2ea069489889d5695"
server_key_iv: "97276dd793289175a127cdb031d21426"
server_key_version: "encryptionkey@0"
I20260812 02:33:01.459415 29228 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260812 02:33:01.464309 29243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:01.466272 29228 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:33:01.466499 29228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "63cd2fa2f56a47a5994eac3ce2713632"
format_stamp: "Formatted at 2026-08-12 02:33:01 on dist-test-slave-vq2q"
server_key: "26222ca9d85856e2ea069489889d5695"
server_key_iv: "97276dd793289175a127cdb031d21426"
server_key_version: "encryptionkey@0"
I20260812 02:33:01.466760 29228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:01.487452 29228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:01.492933 29228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:01.493342 29228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:01.515452 29228 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:42381
I20260812 02:33:01.515472 29308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:42381 every 8 connection(s)
I20260812 02:33:01.517277 29228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:33:01.518781 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 29228
I20260812 02:33:01.519084 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:33:01.520151 12323 external_mini_cluster.cc:1428] Setting key 0c080683f2727cc8c02cbea3a2b77cbf
I20260812 02:33:01.523844 29310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 02:33:01 dist-test-slave-vq2q krb5kdc[29213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:01.541455 29310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632: Bootstrap starting.
I20260812 02:33:01.546445 29310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:01.547991 29310 log.cc:824] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:01.550645 29313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:01.522812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.12.8.254:42381)
0812 02:33:01.523895 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:01.523911 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:01.523975 (+ 64us) server_negotiation.cc:407] Connection header received
0812 02:33:01.526066 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:01.526107 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:01.526663 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:01.527118 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:01.529743 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:01.531439 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:01.533766 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:01.534626 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:01.534801 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:01.541106 (+ 6305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:01.541121 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:01.541143 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:01.541232 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:01.544874 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:01.546327 (+ 1453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:01.546337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:01.546345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:01.546613 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:01.547074 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:01.547078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:01.547080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:01.547792 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:01.548141 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:01.548669 (+ 528us) server_negotiation.cc:299] Negotiation successful
0812 02:33:01.549283 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":485,"threads_started":1}
I20260812 02:33:01.552186 29310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632: No bootstrap required, opened a new log
I20260812 02:33:01.555927 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:33:01.562398 29310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd2fa2f56a47a5994eac3ce2713632" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42381 } }
I20260812 02:33:01.562985 29310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:01.563117 29310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cd2fa2f56a47a5994eac3ce2713632, State: Initialized, Role: FOLLOWER
I20260812 02:33:01.563859 29310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd2fa2f56a47a5994eac3ce2713632" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42381 } }
I20260812 02:33:01.564145 29310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:33:01.564288 29310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:33:01.564472 29310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:01.567494 29310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd2fa2f56a47a5994eac3ce2713632" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42381 } }
I20260812 02:33:01.568248 29310 leader_election.cc:302] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63cd2fa2f56a47a5994eac3ce2713632; no voters:
I20260812 02:33:01.569018 29310 leader_election.cc:288] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:33:01.569152 29315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:01.570127 29315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [term 1 LEADER]: Becoming Leader. State: Replica: 63cd2fa2f56a47a5994eac3ce2713632, State: Running, Role: LEADER
I20260812 02:33:01.570866 29315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd2fa2f56a47a5994eac3ce2713632" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42381 } }
I20260812 02:33:01.571940 29310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:33:01.574399 29317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63cd2fa2f56a47a5994eac3ce2713632. Latest consensus state: current_term: 1 leader_uuid: "63cd2fa2f56a47a5994eac3ce2713632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd2fa2f56a47a5994eac3ce2713632" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42381 } } }
I20260812 02:33:01.575062 29317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:01.574411 29316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63cd2fa2f56a47a5994eac3ce2713632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63cd2fa2f56a47a5994eac3ce2713632" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42381 } } }
I20260812 02:33:01.576009 29316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:01.576207 29321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:33:01.583079 29321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:33:01.595357 29321 catalog_manager.cc:1347] Generated new cluster ID: ca052414d17b4890a79449ce4d43f21a
I20260812 02:33:01.595556 29321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:33:01.617481 29321 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:33:01.619379 29321 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:33:01.635803 29321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63cd2fa2f56a47a5994eac3ce2713632: Generated new TSK 0
I20260812 02:33:01.637001 29321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:33:01.671995 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 29228
I20260812 02:33:01.673892 29300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:01.787122 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 29228
2026-08-12T02:33:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2389 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:33:01 dist-test-slave-vq2q krb5kdc[29340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:33:01 dist-test-slave-vq2q krb5kdc[29340](info): set up 2 sockets
Aug 12 02:33:01 dist-test-slave-vq2q krb5kdc[29340](info): commencing operation
krb5kdc: starting...
W20260812 02:33:02.992266 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.008s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:33:03 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:33:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:33:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:33:03.358225 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33683
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:33683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:03.735184 29356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:03.735503 29356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:03.735637 29356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:03.742985 29356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:33:03.743108 29356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:03.743155 29356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:03.743208 29356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:33:03.743276 29356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:33:03.753515 29356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:33683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:03.754943 29356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:03.756317 29356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:03.769096 29363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:03.769233 29364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:03.769197 29367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:03.770267 29356 server_base.cc:1034] running on GCE node
I20260812 02:33:03.771132 29356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:03.772696 29356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:03.773972 29356 hybrid_clock.cc:648] HybridClock initialized: now 1786501983773870 us; error 79 us; skew 500 ppm
I20260812 02:33:03.774454 29356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:03 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501983, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:03.779106 29356 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:33:03.783114 29356 webserver.cc:460] Webserver started at http://127.12.8.254:34589/ using document root <none> and password file <none>
I20260812 02:33:03.784229 29356 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:03.784420 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:03.784847 29356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:03.788873 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "56d2cb297c55473a9a13ba547e3e22a7"
format_stamp: "Formatted at 2026-08-12 02:33:03 on dist-test-slave-vq2q"
server_key: "0d46e4883bba37b24a2be0c27c650200"
server_key_iv: "825e4b5ab43da26d6adbc2b2887533bd"
server_key_version: "encryptionkey@0"
I20260812 02:33:03.789912 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "56d2cb297c55473a9a13ba547e3e22a7"
format_stamp: "Formatted at 2026-08-12 02:33:03 on dist-test-slave-vq2q"
server_key: "0d46e4883bba37b24a2be0c27c650200"
server_key_iv: "825e4b5ab43da26d6adbc2b2887533bd"
server_key_version: "encryptionkey@0"
I20260812 02:33:03.796752 29356 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260812 02:33:03.801520 29375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:03.803448 29356 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:33:03.803699 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "56d2cb297c55473a9a13ba547e3e22a7"
format_stamp: "Formatted at 2026-08-12 02:33:03 on dist-test-slave-vq2q"
server_key: "0d46e4883bba37b24a2be0c27c650200"
server_key_iv: "825e4b5ab43da26d6adbc2b2887533bd"
server_key_version: "encryptionkey@0"
I20260812 02:33:03.803946 29356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:03.820983 29356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:03.826385 29356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:03.826725 29356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:03.849159 29356 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:33683
I20260812 02:33:03.849233 29437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:33683 every 8 connection(s)
I20260812 02:33:03.851053 29356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:33:03.855406 29438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:03.857509 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 29356
I20260812 02:33:03.857759 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:33:03.858633 12323 external_mini_cluster.cc:1428] Setting key 276ccea211901d986001cae8564f282a
I20260812 02:33:03.869822 29438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Bootstrap starting.
I20260812 02:33:03.873551 29438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:03.874818 29438 log.cc:824] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:33:03 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:03.878718 29438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: No bootstrap required, opened a new log
I20260812 02:33:03.888242 29438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:03.888652 29438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:03.888775 29438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d2cb297c55473a9a13ba547e3e22a7, State: Initialized, Role: FOLLOWER
I20260812 02:33:03.889452 29438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:03.889690 29438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:33:03.889850 29441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:03.861509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.12.8.254:33683)
0812 02:33:03.862543 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:03.862558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:03.862617 (+ 59us) server_negotiation.cc:407] Connection header received
0812 02:33:03.864936 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:03.864978 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:03.865774 (+ 796us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:03.866400 (+ 626us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:03.869234 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:03.871306 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:03.872957 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:03.874058 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:03.874259 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:03.879762 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:03.879773 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:03.879798 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:03.879853 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:03.884103 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:03.885324 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:03.885333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:03.885342 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:03.885593 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:03.886065 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:03.886068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:03.886071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:03.886846 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:03.887180 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:03.887651 (+ 471us) server_negotiation.cc:299] Negotiation successful
0812 02:33:03.888211 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":481,"threads_started":1}
I20260812 02:33:03.889876 29438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:33:03.890110 29438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:03.892896 29438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:03.893517 29438 leader_election.cc:302] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d2cb297c55473a9a13ba547e3e22a7; no voters:
I20260812 02:33:03.894212 29438 leader_election.cc:288] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:33:03.894356 29443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:03.895159 29443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 1 LEADER]: Becoming Leader. State: Replica: 56d2cb297c55473a9a13ba547e3e22a7, State: Running, Role: LEADER
I20260812 02:33:03.895730 29443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:03.896569 29438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:33:03.899219 29445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d2cb297c55473a9a13ba547e3e22a7. Latest consensus state: current_term: 1 leader_uuid: "56d2cb297c55473a9a13ba547e3e22a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } } }
I20260812 02:33:03.899314 29444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d2cb297c55473a9a13ba547e3e22a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } } }
I20260812 02:33:03.899724 29445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:03.899809 29444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:03.906620 29454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:33:03.917732 29454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:33:03.929665 29454 catalog_manager.cc:1347] Generated new cluster ID: 56f9dbc527224550822c1bbda21152a9
I20260812 02:33:03.929809 29454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:33:03.943982 29454 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:33:03.946440 29454 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:33:03.958334 29454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Generated new TSK 0
I20260812 02:33:03.959880 29454 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:33:04.042752 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33683
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:04.489394 29469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:04.489799 29469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:04.489945 29469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:04.498342 29469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:04.498512 29469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:04.498696 29469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:33:04.510499 29469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:04.512281 29469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:04.513953 29469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:04.529709 29479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:04.529798 29476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:04.529769 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:04.530992 29469 server_base.cc:1034] running on GCE node
I20260812 02:33:04.531797 29469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:04.532982 29469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:04.534214 29469 hybrid_clock.cc:648] HybridClock initialized: now 1786501984534137 us; error 71 us; skew 500 ppm
I20260812 02:33:04.534578 29469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:04 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501984, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:04.538343 29469 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:33:04.540248 29469 webserver.cc:460] Webserver started at http://127.12.8.193:36519/ using document root <none> and password file <none>
I20260812 02:33:04.541121 29469 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:04.541311 29469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:04.541693 29469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:04.545399 29469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "250bddab426b4b6a911412ad5eeef234"
format_stamp: "Formatted at 2026-08-12 02:33:04 on dist-test-slave-vq2q"
server_key: "b8db35c22b851794731a1ec0b4ba717a"
server_key_iv: "48fdba4529acfa3b58474761bcd097b5"
server_key_version: "encryptionkey@0"
I20260812 02:33:04.546396 29469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "250bddab426b4b6a911412ad5eeef234"
format_stamp: "Formatted at 2026-08-12 02:33:04 on dist-test-slave-vq2q"
server_key: "b8db35c22b851794731a1ec0b4ba717a"
server_key_iv: "48fdba4529acfa3b58474761bcd097b5"
server_key_version: "encryptionkey@0"
I20260812 02:33:04.553243 29469 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 02:33:04.557859 29487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:04.559708 29469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:33:04.559906 29469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "250bddab426b4b6a911412ad5eeef234"
format_stamp: "Formatted at 2026-08-12 02:33:04 on dist-test-slave-vq2q"
server_key: "b8db35c22b851794731a1ec0b4ba717a"
server_key_iv: "48fdba4529acfa3b58474761bcd097b5"
server_key_version: "encryptionkey@0"
I20260812 02:33:04.560150 29469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:04.581167 29469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:04.586261 29469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:04.586627 29469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:04.587787 29469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:04.589497 29469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:04.589602 29469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:04.589743 29469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:04.589798 29469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:04.627480 29469 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:46521
I20260812 02:33:04.627558 29616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:46521 every 8 connection(s)
I20260812 02:33:04.629124 29469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:33:04.637756 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 29469
I20260812 02:33:04.638024 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:33:04.638809 12323 external_mini_cluster.cc:1428] Setting key 92f11fe801af3dbe593034ea9e905b50
Aug 12 02:33:04 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501984, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:04.653354 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:04.632279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:38207 (local address 127.12.8.254:33683)
0812 02:33:04.632758 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:04.632763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:04.634063 (+ 1300us) server_negotiation.cc:407] Connection header received
0812 02:33:04.636082 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:04.636088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:04.636176 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:04.636278 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:04.638482 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:04.639386 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:04.641380 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:04.641861 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:04.642004 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:04.646627 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:04.646634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:04.646638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:04.646684 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:04.649583 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:04.650578 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:04.650583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:04.650586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:04.650768 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:04.651284 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:04.651290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:04.651292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:04.651658 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:04.651797 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:04.652752 (+ 955us) server_negotiation.cc:299] Negotiation successful
0812 02:33:04.652986 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":246,"threads_started":1}
I20260812 02:33:04.654654 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:04.632922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:38207)
0812 02:33:04.633887 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:04.633934 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:04.635782 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:04.636565 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:04.636574 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:04.637228 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:04.638155 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:04.638185 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:04.639598 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:04.639603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:04.641169 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:04.641215 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:04.642167 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:04.642175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:04.642469 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:04.643304 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:04.643343 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:04.646271 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:04.649777 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:04.649787 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:04.649806 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:04.650400 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:04.650889 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:04.650894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:04.650897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:04.651143 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:04.651795 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:04.651803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:04.652280 (+ 477us) client_negotiation.cc:769] Sending connection context
0812 02:33:04.652614 (+ 334us) client_negotiation.cc:240] Negotiation successful
0812 02:33:04.653213 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":384,"threads_started":1}
I20260812 02:33:04.656491 29617 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33683
I20260812 02:33:04.656980 29617 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:04.658165 29617 heartbeater.cc:511] Master 127.12.8.254:33683 requested a full tablet report, sending...
I20260812 02:33:04.660916 29396 ts_manager.cc:194] Registered new tserver with Master: 250bddab426b4b6a911412ad5eeef234 (127.12.8.193:46521)
I20260812 02:33:04.662588 29396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:38207
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:33:04.718082 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33683
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:05.151489 29627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:05.151816 29627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:05.151961 29627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:05.159560 29627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:05.159706 29627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:05.159866 29627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:33:05.170146 29627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:05.171531 29627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:05.172971 29627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:05.188766 29633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:05.188863 29632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:05.188970 29635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:05.189875 29627 server_base.cc:1034] running on GCE node
I20260812 02:33:05.190558 29627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:05.191560 29627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:05.192814 29627 hybrid_clock.cc:648] HybridClock initialized: now 1786501985192722 us; error 52 us; skew 500 ppm
I20260812 02:33:05.193218 29627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:05 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501985, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:05.196587 29627 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:33:05.198395 29627 webserver.cc:460] Webserver started at http://127.12.8.194:42019/ using document root <none> and password file <none>
I20260812 02:33:05.199138 29627 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:05.199245 29627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:05.199522 29627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:05.202651 29627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "10a089ad5bff44fbb7812fa08360a752"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "63ab828e07b5c6c3113d2c648dc1057e"
server_key_iv: "5ea897b244df734974182369d95ffa3d"
server_key_version: "encryptionkey@0"
I20260812 02:33:05.203518 29627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "10a089ad5bff44fbb7812fa08360a752"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "63ab828e07b5c6c3113d2c648dc1057e"
server_key_iv: "5ea897b244df734974182369d95ffa3d"
server_key_version: "encryptionkey@0"
I20260812 02:33:05.209693 29627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 02:33:05.214069 29644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:05.215916 29627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:33:05.216120 29627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "10a089ad5bff44fbb7812fa08360a752"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "63ab828e07b5c6c3113d2c648dc1057e"
server_key_iv: "5ea897b244df734974182369d95ffa3d"
server_key_version: "encryptionkey@0"
I20260812 02:33:05.216363 29627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:05.238037 29627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:05.242465 29627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:05.242760 29627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:05.243916 29627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:05.245493 29627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:05.245574 29627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:05.245705 29627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:05.245775 29627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:05.282598 29627 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:39459
I20260812 02:33:05.282709 29777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:39459 every 8 connection(s)
I20260812 02:33:05.284003 29627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:33:05.285987 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 29627
I20260812 02:33:05.286221 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:33:05.286700 12323 external_mini_cluster.cc:1428] Setting key 4981a8a42d9fece93b17064ea7eb2f54
Aug 12 02:33:05 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501985, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:05.312772 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.289054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:51199 (local address 127.12.8.254:33683)
0812 02:33:05.289614 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:05.289621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:05.291062 (+ 1441us) server_negotiation.cc:407] Connection header received
0812 02:33:05.293962 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:05.293966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:05.294044 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:05.294159 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:05.296623 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.297513 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.300045 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.300488 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.300594 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.305460 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:05.305466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:05.305470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:05.305525 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:05.308972 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:05.310046 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:05.310052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:05.310056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:05.310222 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:05.310694 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:05.310698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:05.310701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:05.311103 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:05.311284 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:05.312151 (+ 867us) server_negotiation.cc:299] Negotiation successful
0812 02:33:05.312404 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":276,"threads_started":1}
I20260812 02:33:05.314066 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.289789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:51199)
0812 02:33:05.290823 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:05.290896 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:05.293568 (+ 2672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:05.294507 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:05.294521 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:05.295402 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:05.296387 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.296418 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.297709 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.297715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:05.299834 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.299856 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.300769 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.300775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:05.301091 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.301985 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:05.302028 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:05.305107 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:05.309216 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:05.309230 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:05.309253 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:05.309905 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:05.310336 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:05.310339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:05.310342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:05.310565 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:05.311237 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:05.311244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:05.311706 (+ 462us) client_negotiation.cc:769] Sending connection context
0812 02:33:05.311993 (+ 287us) client_negotiation.cc:240] Negotiation successful
0812 02:33:05.312547 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":375,"threads_started":1}
I20260812 02:33:05.315863 29778 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33683
I20260812 02:33:05.316205 29778 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:05.317104 29778 heartbeater.cc:511] Master 127.12.8.254:33683 requested a full tablet report, sending...
I20260812 02:33:05.318923 29396 ts_manager.cc:194] Registered new tserver with Master: 10a089ad5bff44fbb7812fa08360a752 (127.12.8.194:39459)
I20260812 02:33:05.320253 29396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:51199
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:33:05.366577 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33683
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:33:05.666884 29617 heartbeater.cc:503] Master 127.12.8.254:33683 was elected leader, sending a full tablet report...
W20260812 02:33:05.735317 29784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:05.735626 29784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:05.735724 29784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:05.743084 29784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:05.743212 29784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:05.743340 29784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:33:05.753317 29784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:05.754663 29784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:05.755955 29784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:05.770926 29789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:05.771016 29790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:05.770972 29792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:05.772069 29784 server_base.cc:1034] running on GCE node
I20260812 02:33:05.772742 29784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:05.773886 29784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:05.775317 29784 hybrid_clock.cc:648] HybridClock initialized: now 1786501985775239 us; error 80 us; skew 500 ppm
I20260812 02:33:05.775658 29784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:05 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501985, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:05.779551 29784 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:33:05.782256 29784 webserver.cc:460] Webserver started at http://127.12.8.195:34961/ using document root <none> and password file <none>
I20260812 02:33:05.783461 29784 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:05.783672 29784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:05.784175 29784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:05.788338 29784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "a952991dde9d4d96b6f04a389eca4035"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "ebb6d9112d1d0f7e4400a3629d84722f"
server_key_iv: "8de5d9dbb4000e6c481f847509dd1e06"
server_key_version: "encryptionkey@0"
I20260812 02:33:05.789345 29784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "a952991dde9d4d96b6f04a389eca4035"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "ebb6d9112d1d0f7e4400a3629d84722f"
server_key_iv: "8de5d9dbb4000e6c481f847509dd1e06"
server_key_version: "encryptionkey@0"
I20260812 02:33:05.796257 29784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 02:33:05.801373 29801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:05.803599 29784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 02:33:05.803825 29784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "a952991dde9d4d96b6f04a389eca4035"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "ebb6d9112d1d0f7e4400a3629d84722f"
server_key_iv: "8de5d9dbb4000e6c481f847509dd1e06"
server_key_version: "encryptionkey@0"
I20260812 02:33:05.804085 29784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:05.833361 29784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:05.838595 29784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:05.838902 29784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:05.840137 29784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:05.841910 29784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:05.842003 29784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:05.842106 29784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:05.842164 29784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:05.881937 29784 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:45791
I20260812 02:33:05.882019 29938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:45791 every 8 connection(s)
I20260812 02:33:05.883610 29784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:33:05.885073 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 29784
I20260812 02:33:05.885329 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:33:05.885993 12323 external_mini_cluster.cc:1428] Setting key c19cf33b073725546e2a8948b7ae5805
Aug 12 02:33:05 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501985, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:05.907196 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.887062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:53827 (local address 127.12.8.254:33683)
0812 02:33:05.887587 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:05.887593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:05.889024 (+ 1431us) server_negotiation.cc:407] Connection header received
0812 02:33:05.891211 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:05.891216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:05.891312 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:05.891411 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:05.893066 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.893888 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.895438 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.895833 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.895941 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.899971 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:05.899976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:05.899979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:05.900013 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:05.903249 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:05.904391 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:05.904395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:05.904397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:05.904536 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:05.904968 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:05.904971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:05.904973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:05.905314 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:05.905455 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:05.906661 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0812 02:33:05.906906 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":256,"threads_started":1}
I20260812 02:33:05.908936 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.887818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:53827)
0812 02:33:05.888848 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:05.888898 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:05.890940 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:05.891683 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:05.891691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:05.892285 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:05.892901 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.892919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.894038 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.894041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:05.895295 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.895308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.896061 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.896070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:05.896292 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.897048 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:05.897072 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:05.899703 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:05.903489 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:05.903502 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:05.903527 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:05.904245 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:05.904631 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:05.904634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:05.904636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:05.904875 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:05.905500 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:05.905508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:05.906029 (+ 521us) client_negotiation.cc:769] Sending connection context
0812 02:33:05.906543 (+ 514us) client_negotiation.cc:240] Negotiation successful
0812 02:33:05.907390 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":373,"threads_started":1}
I20260812 02:33:05.910899 29939 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33683
I20260812 02:33:05.911257 29939 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:05.912078 29939 heartbeater.cc:511] Master 127.12.8.254:33683 requested a full tablet report, sending...
I20260812 02:33:05.913857 29396 ts_manager.cc:194] Registered new tserver with Master: a952991dde9d4d96b6f04a389eca4035 (127.12.8.195:45791)
I20260812 02:33:05.915063 29396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:53827
I20260812 02:33:05.917405 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:33:05.938828 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.923580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.12.8.254:33683)
0812 02:33:05.923888 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:05.923901 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:05.924280 (+ 379us) server_negotiation.cc:407] Connection header received
0812 02:33:05.924525 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:05.924531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:05.924612 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:05.924713 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:05.926880 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.927711 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.929213 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.929674 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.929810 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.932673 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:05.932679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:05.932682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:05.932737 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:05.935746 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:05.936581 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:05.936587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:05.936590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:05.936748 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:05.937136 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:05.937142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:05.937144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:05.937600 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:05.937894 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:05.938188 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 02:33:05.938455 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 02:33:05.949765 29396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:33:05.957922 29396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:33:06.000357 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.973751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:45791 (local address 127.0.0.1:35880)
0812 02:33:05.989806 (+ 16055us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:05.989828 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:05.989994 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:05.991319 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:05.991326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:05.991362 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:05.992061 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.992075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.994495 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.994506 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:05.997312 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.997328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.999154 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.999161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:05.999382 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.999446 (+ 64us) client_negotiation.cc:769] Sending connection context
0812 02:33:05.999621 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 02:33:05.999861 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15910,"mutex_wait_us":45,"thread_start_us":12005,"threads_started":1}
I20260812 02:33:06.000466 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.973976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.12.8.195:45791)
0812 02:33:05.986034 (+ 12058us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:05.986044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:05.990167 (+ 4123us) server_negotiation.cc:407] Connection header received
0812 02:33:05.990313 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:05.990321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:05.990644 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:05.990891 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:05.992310 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.994290 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.997498 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.998947 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.999097 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.999392 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:05.999802 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.000061 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11864,"thread_start_us":11805,"threads_started":1}
I20260812 02:33:06.001091 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.973631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.12.8.194:39459)
0812 02:33:05.989458 (+ 15827us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:05.989465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:05.989497 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:33:05.989599 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:05.989605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:05.989961 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:05.990835 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:05.992270 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.994076 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.997899 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.999244 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.999405 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.999610 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:06.000038 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.000296 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15647,"thread_start_us":16856,"threads_started":1}
I20260812 02:33:06.001643 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.973408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:39459 (local address 127.0.0.1:33716)
0812 02:33:05.986293 (+ 12885us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:05.986361 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:05.986627 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:05.991174 (+ 4547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:05.991185 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:05.991227 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:05.992012 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.992032 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.994275 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.994282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:05.997682 (+ 3400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.997708 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:05.999565 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:05.999571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:05.999756 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:05.999786 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:33:06.000793 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0812 02:33:06.001009 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12530,"mutex_wait_us":791,"thread_start_us":16047,"threads_started":1}
I20260812 02:33:06.005366 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.990899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46521 (local address 127.0.0.1:47654)
0812 02:33:05.997011 (+ 6112us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:05.997036 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:05.997255 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:05.998055 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:05.998060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:05.998091 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:05.998501 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:05.998512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.000331 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.000337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:06.002436 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:06.002452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.004525 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.004532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:06.004697 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.004735 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 02:33:06.004863 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 02:33:06.004999 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5956,"thread_start_us":5734,"threads_started":1}
I20260812 02:33:06.005499 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:05.990499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.12.8.193:46521)
0812 02:33:05.996559 (+ 6060us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:05.996567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:05.997247 (+ 680us) server_negotiation.cc:407] Connection header received
0812 02:33:05.997510 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:05.997519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:05.997810 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:05.998040 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:05.998710 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.000124 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.002673 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.004265 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.004445 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.004716 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:06.004910 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.005104 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5830,"thread_start_us":5631,"threads_started":1}
I20260812 02:33:06.006083 29700 tablet_service.cc:1467] Processing CreateTablet for tablet 13204fbad5da4b3cb104d98e8d0b45e5 (DEFAULT_TABLE table=test-table [id=157393816f1d46a1867ecfa8fcb20ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:06.006218 29861 tablet_service.cc:1467] Processing CreateTablet for tablet 13204fbad5da4b3cb104d98e8d0b45e5 (DEFAULT_TABLE table=test-table [id=157393816f1d46a1867ecfa8fcb20ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:06.007630 29700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13204fbad5da4b3cb104d98e8d0b45e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:06.008056 29861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13204fbad5da4b3cb104d98e8d0b45e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:06.009819 29543 tablet_service.cc:1467] Processing CreateTablet for tablet 13204fbad5da4b3cb104d98e8d0b45e5 (DEFAULT_TABLE table=test-table [id=157393816f1d46a1867ecfa8fcb20ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:06.011366 29543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13204fbad5da4b3cb104d98e8d0b45e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:06.022904 29963 tablet_bootstrap.cc:492] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Bootstrap starting.
I20260812 02:33:06.026854 29965 tablet_bootstrap.cc:492] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: Bootstrap starting.
I20260812 02:33:06.028757 29966 tablet_bootstrap.cc:492] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Bootstrap starting.
I20260812 02:33:06.028776 29963 tablet_bootstrap.cc:654] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:06.031975 29965 tablet_bootstrap.cc:654] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:06.033038 29966 tablet_bootstrap.cc:654] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:06.033591 29965 log.cc:824] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:06.034507 29966 log.cc:824] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:06.034621 29963 log.cc:824] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:06.038050 29965 tablet_bootstrap.cc:492] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: No bootstrap required, opened a new log
I20260812 02:33:06.040230 29965 ts_tablet_manager.cc:1397] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260812 02:33:06.045279 29963 tablet_bootstrap.cc:492] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: No bootstrap required, opened a new log
I20260812 02:33:06.045709 29963 ts_tablet_manager.cc:1397] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.000s
I20260812 02:33:06.046267 29966 tablet_bootstrap.cc:492] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: No bootstrap required, opened a new log
I20260812 02:33:06.046600 29966 ts_tablet_manager.cc:1397] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20260812 02:33:06.051388 29965 raft_consensus.cc:348] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.051898 29965 raft_consensus.cc:374] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:06.052004 29965 raft_consensus.cc:729] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a089ad5bff44fbb7812fa08360a752, State: Initialized, Role: FOLLOWER
I20260812 02:33:06.052706 29965 consensus_queue.cc:260] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.054239 29778 heartbeater.cc:503] Master 127.12.8.254:33683 was elected leader, sending a full tablet report...
I20260812 02:33:06.054744 29965 ts_tablet_manager.cc:1428] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260812 02:33:06.054436 29963 raft_consensus.cc:348] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.054999 29963 raft_consensus.cc:374] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:06.055117 29963 raft_consensus.cc:729] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a952991dde9d4d96b6f04a389eca4035, State: Initialized, Role: FOLLOWER
I20260812 02:33:06.055904 29963 consensus_queue.cc:260] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.056802 29966 raft_consensus.cc:348] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.057297 29966 raft_consensus.cc:374] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:06.057408 29966 raft_consensus.cc:729] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250bddab426b4b6a911412ad5eeef234, State: Initialized, Role: FOLLOWER
I20260812 02:33:06.058128 29966 consensus_queue.cc:260] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.058861 29963 ts_tablet_manager.cc:1428] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260812 02:33:06.059644 29939 heartbeater.cc:503] Master 127.12.8.254:33683 was elected leader, sending a full tablet report...
I20260812 02:33:06.060828 29966 ts_tablet_manager.cc:1428] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260812 02:33:06.112826 29975 raft_consensus.cc:482] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:33:06.113216 29975 raft_consensus.cc:504] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.115163 29975 leader_election.cc:288] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10a089ad5bff44fbb7812fa08360a752 (127.12.8.194:39459), 250bddab426b4b6a911412ad5eeef234 (127.12.8.193:46521)
I20260812 02:33:06.122748 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:06.115629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:39459 (local address 127.12.8.195:40111)
0812 02:33:06.115880 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:06.115902 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:06.116049 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:06.116507 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:06.116513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:06.116567 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:06.116922 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:06.116945 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.118369 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.118374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:06.120422 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:06.120445 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.122010 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.122016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:06.122189 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.122224 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 02:33:06.122314 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:33:06.122450 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":81}
I20260812 02:33:06.122879 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:06.115782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:40111 (local address 127.12.8.194:39459)
0812 02:33:06.116018 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:06.116025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:06.116058 (+ 33us) server_negotiation.cc:407] Connection header received
0812 02:33:06.116214 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:06.116220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:06.116308 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:06.116442 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:06.117168 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.118201 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.120681 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.121816 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.122023 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.122110 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:06.122372 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.122561 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:33:06.123836 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:06.116540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:56523 (local address 127.12.8.193:46521)
0812 02:33:06.116796 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:06.116812 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:06.117459 (+ 647us) server_negotiation.cc:407] Connection header received
0812 02:33:06.117616 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:06.117621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:06.117712 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:06.117840 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:06.118431 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.119184 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.121604 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.122757 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.122953 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.123035 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:06.123466 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.123626 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 02:33:06.123945 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:06.116123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46521 (local address 127.12.8.195:56523)
0812 02:33:06.117342 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:06.117361 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:06.117482 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:06.117850 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:06.117853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:06.117871 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:06.118245 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:06.118254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.119371 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.119377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:06.121425 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:06.121439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.123018 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.123025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:06.123204 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.123240 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 02:33:06.123350 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 02:33:06.123507 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":28,"thread_start_us":720,"threads_started":1}
I20260812 02:33:06.124866 29723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13204fbad5da4b3cb104d98e8d0b45e5" candidate_uuid: "a952991dde9d4d96b6f04a389eca4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a089ad5bff44fbb7812fa08360a752" is_pre_election: true
I20260812 02:33:06.125542 29565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13204fbad5da4b3cb104d98e8d0b45e5" candidate_uuid: "a952991dde9d4d96b6f04a389eca4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250bddab426b4b6a911412ad5eeef234" is_pre_election: true
I20260812 02:33:06.125849 29723 raft_consensus.cc:2454] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a952991dde9d4d96b6f04a389eca4035 in term 0.
I20260812 02:33:06.126330 29565 raft_consensus.cc:2454] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a952991dde9d4d96b6f04a389eca4035 in term 0.
I20260812 02:33:06.126966 29803 leader_election.cc:302] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10a089ad5bff44fbb7812fa08360a752, a952991dde9d4d96b6f04a389eca4035; no voters:
I20260812 02:33:06.127609 29975 raft_consensus.cc:2781] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:33:06.127835 29975 raft_consensus.cc:482] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:33:06.127949 29975 raft_consensus.cc:3037] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:06.131141 29975 raft_consensus.cc:504] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.132016 29975 leader_election.cc:288] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [CANDIDATE]: Term 1 election: Requested vote from peers 10a089ad5bff44fbb7812fa08360a752 (127.12.8.194:39459), 250bddab426b4b6a911412ad5eeef234 (127.12.8.193:46521)
W20260812 02:33:06.132802 29618 tablet.cc:2369] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:06.132870 29723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13204fbad5da4b3cb104d98e8d0b45e5" candidate_uuid: "a952991dde9d4d96b6f04a389eca4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a089ad5bff44fbb7812fa08360a752"
I20260812 02:33:06.133114 29565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13204fbad5da4b3cb104d98e8d0b45e5" candidate_uuid: "a952991dde9d4d96b6f04a389eca4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250bddab426b4b6a911412ad5eeef234"
I20260812 02:33:06.133232 29723 raft_consensus.cc:3037] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:06.133447 29565 raft_consensus.cc:3037] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Advancing to term 1
W20260812 02:33:06.135759 29940 tablet.cc:2369] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:06.137279 29565 raft_consensus.cc:2454] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a952991dde9d4d96b6f04a389eca4035 in term 1.
I20260812 02:33:06.137442 29723 raft_consensus.cc:2454] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a952991dde9d4d96b6f04a389eca4035 in term 1.
I20260812 02:33:06.138010 29803 leader_election.cc:302] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250bddab426b4b6a911412ad5eeef234, a952991dde9d4d96b6f04a389eca4035; no voters:
I20260812 02:33:06.138403 29975 raft_consensus.cc:2781] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:06.139014 29975 raft_consensus.cc:686] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 1 LEADER]: Becoming Leader. State: Replica: a952991dde9d4d96b6f04a389eca4035, State: Running, Role: LEADER
I20260812 02:33:06.139678 29975 consensus_queue.cc:237] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } }
I20260812 02:33:06.146070 29395 catalog_manager.cc:5495] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 reported cstate change: term changed from 0 to 1, leader changed from <none> to a952991dde9d4d96b6f04a389eca4035 (127.12.8.195). New cstate: current_term: 1 leader_uuid: "a952991dde9d4d96b6f04a389eca4035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } health_report { overall_health: UNKNOWN } } }
I20260812 02:33:06.190984 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:06.184845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.12.8.195:45791)
0812 02:33:06.185097 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:06.185103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:06.185125 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:33:06.185283 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:06.185287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:06.185396 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:06.185580 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:33:06.186064 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.186862 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.188402 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.188869 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.188985 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.189561 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:33:06.190303 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:33:06.190438 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:06.190608 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.190705 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 02:33:06.201275 29565 raft_consensus.cc:1261] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 1 FOLLOWER]: Refusing update from remote peer a952991dde9d4d96b6f04a389eca4035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:06.201398 29723 raft_consensus.cc:1261] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 1 FOLLOWER]: Refusing update from remote peer a952991dde9d4d96b6f04a389eca4035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:06.202427 29978 consensus_queue.cc:1035] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:06.202898 29975 consensus_queue.cc:1035] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:06.222154 29983 mvcc.cc:205] Tried to move back new op lower bound from 7317512135467827200 to 7317512135232974848. Current Snapshot: MvccSnapshot[applied={T|T < 7317512135467827200 or (T in {7317512135467827200})}]
I20260812 02:33:06.227583 29982 mvcc.cc:205] Tried to move back new op lower bound from 7317512135467827200 to 7317512135232974848. Current Snapshot: MvccSnapshot[applied={T|T < 7317512135467827200}]
I20260812 02:33:06.239084 29395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:06.239440 29395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:06.244825 29395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Sending DeleteTablet for 3 replicas of tablet 13204fbad5da4b3cb104d98e8d0b45e5
I20260812 02:33:06.246814 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 29469
I20260812 02:33:06.246655 29700 tablet_service.cc:1514] Processing DeleteTablet for tablet 13204fbad5da4b3cb104d98e8d0b45e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:06 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33716
I20260812 02:33:06.246886 29543 tablet_service.cc:1514] Processing DeleteTablet for tablet 13204fbad5da4b3cb104d98e8d0b45e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:06 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:47654
I20260812 02:33:06.247380 29861 tablet_service.cc:1514] Processing DeleteTablet for tablet 13204fbad5da4b3cb104d98e8d0b45e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:06 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:35880
I20260812 02:33:06.261525 29998 tablet_replica.cc:330] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: stopping tablet replica
I20260812 02:33:06.262099 29998 raft_consensus.cc:2229] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:33:06.262954 29998 raft_consensus.cc:2258] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:06.271374 29998 ts_tablet_manager.cc:1905] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:06.278190 29995 tablet_replica.cc:330] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: stopping tablet replica
I20260812 02:33:06.278759 29995 raft_consensus.cc:2229] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:06.279124 29996 tablet_replica.cc:330] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: stopping tablet replica
I20260812 02:33:06.279354 29995 raft_consensus.cc:2258] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:06.279660 29996 raft_consensus.cc:2229] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:06.280280 29996 raft_consensus.cc:2258] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:06.282721 29996 ts_tablet_manager.cc:1905] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 02:33:06 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:33:06.284178 29995 ts_tablet_manager.cc:1905] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:06.289814 29998 ts_tablet_manager.cc:1918] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:06.290199 29998 log.cc:1196] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/13204fbad5da4b3cb104d98e8d0b45e5
I20260812 02:33:06.290464 29996 ts_tablet_manager.cc:1918] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:06.290684 29996 log.cc:1196] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/13204fbad5da4b3cb104d98e8d0b45e5
I20260812 02:33:06.291193 29998 ts_tablet_manager.cc:1939] T 13204fbad5da4b3cb104d98e8d0b45e5 P a952991dde9d4d96b6f04a389eca4035: Deleting consensus metadata
I20260812 02:33:06.291296 29996 ts_tablet_manager.cc:1939] T 13204fbad5da4b3cb104d98e8d0b45e5 P 250bddab426b4b6a911412ad5eeef234: Deleting consensus metadata
I20260812 02:33:06.293735 29377 catalog_manager.cc:4841] TS 250bddab426b4b6a911412ad5eeef234 (127.12.8.193:46521): tablet 13204fbad5da4b3cb104d98e8d0b45e5 (table test-table [id=157393816f1d46a1867ecfa8fcb20ad1]) successfully deleted
I20260812 02:33:06.294112 29378 catalog_manager.cc:4841] TS a952991dde9d4d96b6f04a389eca4035 (127.12.8.195:45791): tablet 13204fbad5da4b3cb104d98e8d0b45e5 (table test-table [id=157393816f1d46a1867ecfa8fcb20ad1]) successfully deleted
I20260812 02:33:06.298203 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:06.248055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.12.8.193:46521)
0812 02:33:06.248281 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:06.248288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:06.257979 (+ 9691us) server_negotiation.cc:407] Connection header received
0812 02:33:06.258544 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:06.258552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:06.258695 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:06.258774 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:06.263463 (+ 4689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.264768 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.267073 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.267807 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.267918 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.287962 (+ 20044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:06.287994 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:06.288013 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:06.288083 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:06.292357 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:06.293724 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:06.293737 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:06.293741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:06.294083 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:06.294861 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:06.294866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:06.294870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:06.295458 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:06.295850 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:06.297375 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.297689 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:33:06.299080 29995 ts_tablet_manager.cc:1918] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:06.299327 29995 log.cc:1196] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/13204fbad5da4b3cb104d98e8d0b45e5
I20260812 02:33:06.299887 29611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:06.299999 29995 ts_tablet_manager.cc:1939] T 13204fbad5da4b3cb104d98e8d0b45e5 P 10a089ad5bff44fbb7812fa08360a752: Deleting consensus metadata
I20260812 02:33:06.303094 29377 catalog_manager.cc:4841] TS 10a089ad5bff44fbb7812fa08360a752 (127.12.8.194:39459): tablet 13204fbad5da4b3cb104d98e8d0b45e5 (table test-table [id=157393816f1d46a1867ecfa8fcb20ad1]) successfully deleted
I20260812 02:33:06.410099 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 29469
I20260812 02:33:06.441583 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 29627
Aug 12 02:33:06 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:33:06.463740 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:06.442653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.12.8.194:39459)
0812 02:33:06.442902 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:06.442909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:06.442937 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:33:06.443020 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:06.443025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:06.443121 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:06.443236 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:06.444681 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.445635 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.447083 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.447544 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.447663 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.456628 (+ 8965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:06.456654 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:06.456676 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:06.456737 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:06.460133 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:06.461228 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:06.461243 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:06.461246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:06.461448 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:06.461909 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:06.461913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:06.461917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:06.462463 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:06.462755 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:06.463135 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.463372 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 02:33:06.464602 29773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:06.567785 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 29627
I20260812 02:33:06.601158 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 29784
Aug 12 02:33:06 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:33:06.622076 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:06.602360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.12.8.195:45791)
0812 02:33:06.602649 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:06.602656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:06.602682 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:33:06.602833 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:06.602837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:06.602961 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:06.603074 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:06.604628 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.605679 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.607168 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:06.607688 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:06.607819 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:06.614123 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:06.614150 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:06.614173 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:06.614244 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:06.618074 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:06.619266 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:06.619277 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:06.619279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:06.619479 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:06.620007 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:06.620011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:06.620014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:06.620536 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:06.620863 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:06.621335 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 02:33:06.621586 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 02:33:06.623085 29932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:06.775851 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 29784
I20260812 02:33:06.815811 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 29356
I20260812 02:33:06.817399 29433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:06.951012 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 29356
I20260812 02:33:06.992206 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33683
--webserver_interface=127.12.8.254
--webserver_port=34589
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:33683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:07.384210 30005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:07.384589 30005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:07.384706 30005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:07.392203 30005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:33:07.392334 30005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:07.392382 30005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:07.392469 30005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:33:07.392535 30005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:33:07.402887 30005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:33683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=34589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:07.404357 30005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:07.405803 30005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:07.436336 30011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:07.436357 30012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:07.436434 30014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:07.437278 30005 server_base.cc:1034] running on GCE node
I20260812 02:33:07.438548 30005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:07.440405 30005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:07.441798 30005 hybrid_clock.cc:648] HybridClock initialized: now 1786501987441721 us; error 56 us; skew 500 ppm
I20260812 02:33:07.442268 30005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:07 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501987, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:07.446360 30005 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:33:07.448451 30005 webserver.cc:460] Webserver started at http://127.12.8.254:34589/ using document root <none> and password file <none>
I20260812 02:33:07.449353 30005 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:07.449518 30005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:07.456485 30005 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:33:07.460114 30021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:07.461894 30005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:33:07.462090 30005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "56d2cb297c55473a9a13ba547e3e22a7"
format_stamp: "Formatted at 2026-08-12 02:33:03 on dist-test-slave-vq2q"
server_key: "0d46e4883bba37b24a2be0c27c650200"
server_key_iv: "825e4b5ab43da26d6adbc2b2887533bd"
server_key_version: "encryptionkey@0"
I20260812 02:33:07.463251 30005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:07.484117 30005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:07.489157 30005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:07.489500 30005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:07.510787 30005 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:33683
I20260812 02:33:07.510772 30083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:33683 every 8 connection(s)
I20260812 02:33:07.512863 30005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:33:07.515097 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 30005
I20260812 02:33:07.516028 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:46521
--local_ip_for_outbound_sockets=127.12.8.193
--tserver_master_addrs=127.12.8.254:33683
--webserver_port=36519
--webserver_interface=127.12.8.193
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:33:07.520982 30084 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:33:07.525303 30084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Bootstrap starting.
I20260812 02:33:07.546808 30084 log.cc:824] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:07.559583 30084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:33:07.560134 30084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Bootstrap complete.
I20260812 02:33:07.570096 30084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:07.571007 30084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d2cb297c55473a9a13ba547e3e22a7, State: Initialized, Role: FOLLOWER
I20260812 02:33:07.571671 30084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:07.571923 30084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:33:07.572039 30084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:33:07.572201 30084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:33:07.576845 30084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:07.577512 30084 leader_election.cc:302] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d2cb297c55473a9a13ba547e3e22a7; no voters:
I20260812 02:33:07.578167 30084 leader_election.cc:288] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:33:07.578316 30090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:33:07.579799 30090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 2 LEADER]: Becoming Leader. State: Replica: 56d2cb297c55473a9a13ba547e3e22a7, State: Running, Role: LEADER
I20260812 02:33:07.580428 30090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:07.580879 30084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:33:07.583694 30092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d2cb297c55473a9a13ba547e3e22a7. Latest consensus state: current_term: 2 leader_uuid: "56d2cb297c55473a9a13ba547e3e22a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } } }
I20260812 02:33:07.583742 30091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "56d2cb297c55473a9a13ba547e3e22a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } } }
I20260812 02:33:07.584342 30091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:07.584350 30092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:07.597925 30097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:33:07.609169 30097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:33:07.610920 30097 catalog_manager.cc:1259] Loaded cluster ID: 56f9dbc527224550822c1bbda21152a9
I20260812 02:33:07.611049 30097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:33:07.614856 30097 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 02:33:07.632774 30097 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 02:33:07.657516 30097 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:33:07.659969 30097 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Loaded TSK: 0
I20260812 02:33:07.662499 30097 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:33:07.988767 30087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:07.989156 30087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:07.989334 30087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:07.997689 30087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:07.997865 30087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:07.998062 30087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:33:08.009894 30087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:46521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=36519
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:08.011655 30087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:08.013387 30087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:08.029496 30117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:08.029433 30119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:08.029795 30121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:08.030552 30087 server_base.cc:1034] running on GCE node
I20260812 02:33:08.031272 30087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:08.032425 30087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:08.033646 30087 hybrid_clock.cc:648] HybridClock initialized: now 1786501988033593 us; error 45 us; skew 500 ppm
I20260812 02:33:08.033995 30087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:08 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501988, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:08.037484 30087 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:33:08.039188 30087 webserver.cc:460] Webserver started at http://127.12.8.193:36519/ using document root <none> and password file <none>
I20260812 02:33:08.039976 30087 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:08.040118 30087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:08.046337 30087 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:33:08.049697 30129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:08.051438 30087 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 02:33:08.051630 30087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "250bddab426b4b6a911412ad5eeef234"
format_stamp: "Formatted at 2026-08-12 02:33:04 on dist-test-slave-vq2q"
server_key: "b8db35c22b851794731a1ec0b4ba717a"
server_key_iv: "48fdba4529acfa3b58474761bcd097b5"
server_key_version: "encryptionkey@0"
I20260812 02:33:08.052727 30087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:08.070220 30087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:08.074760 30087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:08.075039 30087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:08.076170 30087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:08.077764 30087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:08.077865 30087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:08.077978 30087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:08.078054 30087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:08.116004 30087 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:46521
I20260812 02:33:08.116114 30260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:46521 every 8 connection(s)
I20260812 02:33:08.117647 30087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:33:08.121140 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 30087
I20260812 02:33:08.122066 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:39459
--local_ip_for_outbound_sockets=127.12.8.194
--tserver_master_addrs=127.12.8.254:33683
--webserver_port=42019
--webserver_interface=127.12.8.194
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:33:08 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501988, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:08.149634 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:08.121276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:32991 (local address 127.12.8.254:33683)
0812 02:33:08.122059 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:08.122080 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:08.126319 (+ 4239us) server_negotiation.cc:407] Connection header received
0812 02:33:08.129522 (+ 3203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:08.129539 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:08.130000 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:08.130365 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:08.132786 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:08.134200 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:08.135898 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:08.136576 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:08.136771 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:08.140772 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:08.140784 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:08.140804 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:08.140861 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:08.144106 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:08.145138 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:08.145146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:08.145153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:08.145418 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:08.145973 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:08.145976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:08.145978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:08.146617 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:08.146793 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:08.147871 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0812 02:33:08.148335 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":317,"threads_started":1}
I20260812 02:33:08.149869 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:08.121367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:32991)
0812 02:33:08.126026 (+ 4659us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:08.126114 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:08.128737 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:08.130675 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:08.130689 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:08.131584 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:08.132509 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:08.132536 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:08.134442 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:08.134448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:08.135733 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:08.135753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:08.136854 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:08.136862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:08.137138 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:08.137866 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:08.137890 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:08.140390 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:08.144313 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:08.144323 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:08.144340 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:08.144990 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:08.145563 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:08.145568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:08.145571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:08.145853 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:08.146800 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:08.146808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:08.147323 (+ 515us) client_negotiation.cc:769] Sending connection context
0812 02:33:08.147653 (+ 330us) client_negotiation.cc:240] Negotiation successful
0812 02:33:08.148209 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4212,"thread_start_us":3931,"threads_started":1}
I20260812 02:33:08.153918 30261 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33683
I20260812 02:33:08.154464 30261 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:08.155464 30261 heartbeater.cc:511] Master 127.12.8.254:33683 requested a full tablet report, sending...
I20260812 02:33:08.158331 30043 ts_manager.cc:194] Registered new tserver with Master: 250bddab426b4b6a911412ad5eeef234 (127.12.8.193:46521)
I20260812 02:33:08.160099 30043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:32991
W20260812 02:33:08.509874 30265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:08.510221 30265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:08.510358 30265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:08.517903 30265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:08.518049 30265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:08.518203 30265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:33:08.528469 30265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:39459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=42019
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:08.529955 30265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:08.531404 30265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:08.545800 30271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:08.545771 30277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:08.546629 30270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:08.552542 30265 server_base.cc:1034] running on GCE node
I20260812 02:33:08.553530 30265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:08.554877 30265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:08.556108 30265 hybrid_clock.cc:648] HybridClock initialized: now 1786501988556020 us; error 64 us; skew 500 ppm
I20260812 02:33:08.556507 30265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:08 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501988, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:08.560611 30265 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:33:08.562626 30265 webserver.cc:460] Webserver started at http://127.12.8.194:42019/ using document root <none> and password file <none>
I20260812 02:33:08.563560 30265 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:08.563709 30265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:08.570942 30265 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 02:33:08.574708 30286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:08.576705 30265 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 02:33:08.576936 30265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "10a089ad5bff44fbb7812fa08360a752"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "63ab828e07b5c6c3113d2c648dc1057e"
server_key_iv: "5ea897b244df734974182369d95ffa3d"
server_key_version: "encryptionkey@0"
I20260812 02:33:08.578261 30265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:08.622967 30265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:08.628374 30265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:08.628707 30265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:08.630105 30265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:08.631953 30265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:08.632067 30265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:08.632186 30265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:08.632263 30265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:08.670562 30265 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:39459
I20260812 02:33:08.670639 30428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:39459 every 8 connection(s)
I20260812 02:33:08.672384 30265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:33:08.674914 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 30265
I20260812 02:33:08.675676 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:45791
--local_ip_for_outbound_sockets=127.12.8.195
--tserver_master_addrs=127.12.8.254:33683
--webserver_port=34961
--webserver_interface=127.12.8.195
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:33:08 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501988, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:08.700446 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:08.675700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:53129 (local address 127.12.8.254:33683)
0812 02:33:08.681705 (+ 6005us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:08.681711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:08.681738 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:33:08.681876 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:08.681884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:08.681984 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:08.682165 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:08.684725 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:08.685645 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:08.687742 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:08.688306 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:08.688458 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:08.693385 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:08.693392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:08.693396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:08.693452 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:08.696408 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:08.697521 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:08.697526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:08.697528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:08.697717 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:08.698272 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:08.698277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:08.698279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:08.698635 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:08.698781 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:08.699722 (+ 941us) server_negotiation.cc:299] Negotiation successful
0812 02:33:08.700017 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5845,"thread_start_us":5672,"threads_started":1}
I20260812 02:33:08.701887 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:08.676456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:53129)
0812 02:33:08.677484 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:08.677569 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:08.679925 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:08.682577 (+ 2652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:08.682589 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:08.683489 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:08.684434 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:08.684463 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:08.685892 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:08.685897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:08.687511 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:08.687533 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:08.688628 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:08.688637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:08.688952 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:08.689893 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:08.689925 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:08.692967 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:08.696658 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:08.696670 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:08.696692 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:08.697366 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:08.697856 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:08.697859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:08.697862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:08.698136 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:08.698795 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:08.698806 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:08.699273 (+ 467us) client_negotiation.cc:769] Sending connection context
0812 02:33:08.699575 (+ 302us) client_negotiation.cc:240] Negotiation successful
0812 02:33:08.700315 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":395,"threads_started":1}
I20260812 02:33:08.704161 30430 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33683
I20260812 02:33:08.704603 30430 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:08.705641 30430 heartbeater.cc:511] Master 127.12.8.254:33683 requested a full tablet report, sending...
I20260812 02:33:08.707772 30043 ts_manager.cc:194] Registered new tserver with Master: 10a089ad5bff44fbb7812fa08360a752 (127.12.8.194:39459)
I20260812 02:33:08.708707 30043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:53129
W20260812 02:33:09.028119 30434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:09.028431 30434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:09.028568 30434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:09.036342 30434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:09.036491 30434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:09.036649 30434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:33:09.046823 30434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:45791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=34961
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:09.048198 30434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:09.049647 30434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:09.064738 30439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:09.064785 30440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:09.064937 30443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:09.071467 30434 server_base.cc:1034] running on GCE node
I20260812 02:33:09.072614 30434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:09.074108 30434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:09.075359 30434 hybrid_clock.cc:648] HybridClock initialized: now 1786501989075282 us; error 69 us; skew 500 ppm
I20260812 02:33:09.075702 30434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:09 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501989, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:09.079279 30434 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:33:09.081027 30434 webserver.cc:460] Webserver started at http://127.12.8.195:34961/ using document root <none> and password file <none>
I20260812 02:33:09.081797 30434 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:09.081933 30434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:09.087966 30434 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 02:33:09.091248 30450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:09.092937 30434 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 02:33:09.093101 30434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "a952991dde9d4d96b6f04a389eca4035"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "ebb6d9112d1d0f7e4400a3629d84722f"
server_key_iv: "8de5d9dbb4000e6c481f847509dd1e06"
server_key_version: "encryptionkey@0"
I20260812 02:33:09.094167 30434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:09.118861 30434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:09.123719 30434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:09.123994 30434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:09.125118 30434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:09.126776 30434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:09.126874 30434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:09.126994 30434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:09.127069 30434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:09.162124 30434 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:45791
I20260812 02:33:09.162334 30589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:45791 every 8 connection(s)
I20260812 02:33:09.163686 30434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:33:09.164209 30261 heartbeater.cc:503] Master 127.12.8.254:33683 was elected leader, sending a full tablet report...
I20260812 02:33:09.170189 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 30434
Aug 12 02:33:09 dist-test-slave-vq2q krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501989, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:09.189035 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.166651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:38199 (local address 127.12.8.254:33683)
0812 02:33:09.166901 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.166907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.168599 (+ 1692us) server_negotiation.cc:407] Connection header received
0812 02:33:09.170556 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.170562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.170689 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.170830 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:09.172828 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.173696 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.176007 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.176709 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.176860 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.181258 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:09.181267 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:09.181273 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:09.181325 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:09.185267 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.186193 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.186199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.186203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.186375 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.186797 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.186803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.186805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.187216 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:09.187385 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.188302 (+ 917us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.188628 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":89}
I20260812 02:33:09.189734 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.171399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.12.8.254:33683)
0812 02:33:09.171979 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.171985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.172314 (+ 329us) server_negotiation.cc:407] Connection header received
0812 02:33:09.172497 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.172500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.172557 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.172652 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:09.174531 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.175824 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.177822 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.178506 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.178668 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.182051 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:09.182061 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:09.182064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:09.182111 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:09.185709 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.186837 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.186841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.186844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.187010 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.187467 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.187471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.187474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.187890 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:09.188172 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.188641 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.188890 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":440,"thread_start_us":282,"threads_started":1}
I20260812 02:33:09.190043 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.167457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:38199)
0812 02:33:09.168406 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:09.168461 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:09.170227 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:09.171117 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:09.171129 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:09.171945 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:09.172599 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.172617 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.173901 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.173905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.175791 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.175808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.176983 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.176989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.177236 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.178037 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:09.178115 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:09.180910 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:09.185446 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:09.185456 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:09.185472 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:09.186046 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:09.186492 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:09.186495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:09.186497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:09.186679 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:09.187350 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:09.187356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:09.187794 (+ 438us) client_negotiation.cc:769] Sending connection context
0812 02:33:09.188122 (+ 328us) client_negotiation.cc:240] Negotiation successful
0812 02:33:09.188638 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":364,"threads_started":1}
I20260812 02:33:09.191591 30590 heartbeater.cc:347] Connected to a master server at 127.12.8.254:33683
I20260812 02:33:09.191890 30590 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:09.192616 30590 heartbeater.cc:511] Master 127.12.8.254:33683 requested a full tablet report, sending...
I20260812 02:33:09.194152 30043 ts_manager.cc:194] Registered new tserver with Master: a952991dde9d4d96b6f04a389eca4035 (127.12.8.195:45791)
I20260812 02:33:09.195086 30043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:38199
I20260812 02:33:09.205731 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:33:09.213557 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.206939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.12.8.254:33683)
0812 02:33:09.207273 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.207281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.207798 (+ 517us) server_negotiation.cc:407] Connection header received
0812 02:33:09.208015 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.208019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.208191 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.208346 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:33:09.208784 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.209562 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.211255 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.211676 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.211820 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.212352 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:33:09.212933 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:33:09.213050 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.213142 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.213256 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 02:33:09.214687 30043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:33:09.222463 30043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:33:09.250067 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.237416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.12.8.193:46521)
0812 02:33:09.238791 (+ 1375us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.238800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.240296 (+ 1496us) server_negotiation.cc:407] Connection header received
0812 02:33:09.240732 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.240741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.241028 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.241251 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:09.242423 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.243964 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.246863 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.248356 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.248476 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.248742 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.249513 (+ 771us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.249702 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":1005,"threads_started":1}
I20260812 02:33:09.250954 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.237290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46521 (local address 127.0.0.1:47678)
0812 02:33:09.240107 (+ 2817us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:09.240155 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:09.240466 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:09.241647 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:09.241654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:09.241693 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:09.242182 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.242196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.244144 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.244149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.246676 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.246690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.248945 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.248951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.249171 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.249279 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 02:33:09.250327 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0812 02:33:09.250533 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2555,"thread_start_us":2413,"threads_started":1}
I20260812 02:33:09.252674 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.237416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:45791 (local address 127.0.0.1:35912)
0812 02:33:09.241769 (+ 4353us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:09.241788 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:09.241915 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:09.242946 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:09.242951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:09.242977 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:09.243317 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.243328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.246487 (+ 3159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.246493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.249857 (+ 3364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.249873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.251928 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.251936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.252102 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.252137 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 02:33:09.252230 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 02:33:09.252371 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4216,"thread_start_us":1450,"threads_started":1}
I20260812 02:33:09.253702 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.238225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.12.8.195:45791)
0812 02:33:09.239453 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.239461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.242058 (+ 2597us) server_negotiation.cc:407] Connection header received
0812 02:33:09.242182 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.242189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.242446 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.242632 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:09.243528 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.246283 (+ 2755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.250060 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.251587 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.251726 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.251990 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.252692 (+ 702us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.253298 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":828,"threads_started":1}
I20260812 02:33:09.257319 30183 tablet_service.cc:1467] Processing CreateTablet for tablet 8fad0d9aaed3499ab711a1209a2f6e05 (DEFAULT_TABLE table=test-table [id=70b416a44ede41f2a8ba501c53bb72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:09.257697 30510 tablet_service.cc:1467] Processing CreateTablet for tablet 8fad0d9aaed3499ab711a1209a2f6e05 (DEFAULT_TABLE table=test-table [id=70b416a44ede41f2a8ba501c53bb72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:09.259297 30510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fad0d9aaed3499ab711a1209a2f6e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:09.259307 30183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fad0d9aaed3499ab711a1209a2f6e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:09.260983 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.240166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:39459 (local address 127.0.0.1:33720)
0812 02:33:09.251193 (+ 11027us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:09.251213 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:09.251329 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:09.253459 (+ 2130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:09.253464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:09.253493 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:09.253677 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.253688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.255829 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.255837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.257985 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.258010 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.260098 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.260105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.260315 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.260361 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 02:33:09.260447 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 02:33:09.260631 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10921,"thread_start_us":11448,"threads_started":1}
I20260812 02:33:09.262619 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.240524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.12.8.194:39459)
0812 02:33:09.252492 (+ 11968us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.252502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.252534 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:33:09.252658 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.252668 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.252981 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.253874 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:09.254117 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.255613 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.258288 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.259852 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.261661 (+ 1809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.261928 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.262050 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.262277 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11767,"thread_start_us":11669,"threads_started":1}
I20260812 02:33:09.266942 30346 tablet_service.cc:1467] Processing CreateTablet for tablet 8fad0d9aaed3499ab711a1209a2f6e05 (DEFAULT_TABLE table=test-table [id=70b416a44ede41f2a8ba501c53bb72e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:09.268955 30346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fad0d9aaed3499ab711a1209a2f6e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:09.275211 30608 tablet_bootstrap.cc:492] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Bootstrap starting.
I20260812 02:33:09.280367 30608 tablet_bootstrap.cc:654] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:09.281081 30609 tablet_bootstrap.cc:492] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Bootstrap starting.
I20260812 02:33:09.281987 30608 log.cc:824] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:09.284684 30611 tablet_bootstrap.cc:492] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Bootstrap starting.
I20260812 02:33:09.285647 30609 tablet_bootstrap.cc:654] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:09.286248 30608 tablet_bootstrap.cc:492] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: No bootstrap required, opened a new log
I20260812 02:33:09.286592 30608 ts_tablet_manager.cc:1397] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260812 02:33:09.287226 30609 log.cc:824] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:09.288590 30611 tablet_bootstrap.cc:654] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:09.290071 30611 log.cc:824] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:09.290448 30609 tablet_bootstrap.cc:492] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: No bootstrap required, opened a new log
I20260812 02:33:09.290773 30609 ts_tablet_manager.cc:1397] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260812 02:33:09.293857 30611 tablet_bootstrap.cc:492] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: No bootstrap required, opened a new log
I20260812 02:33:09.294207 30611 ts_tablet_manager.cc:1397] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260812 02:33:09.297713 30608 raft_consensus.cc:348] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.298660 30608 raft_consensus.cc:374] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:09.298457 30609 raft_consensus.cc:348] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.298836 30608 raft_consensus.cc:729] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a952991dde9d4d96b6f04a389eca4035, State: Initialized, Role: FOLLOWER
I20260812 02:33:09.298941 30609 raft_consensus.cc:374] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:09.299041 30609 raft_consensus.cc:729] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250bddab426b4b6a911412ad5eeef234, State: Initialized, Role: FOLLOWER
I20260812 02:33:09.299629 30609 consensus_queue.cc:260] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.299952 30608 consensus_queue.cc:260] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.306247 30590 heartbeater.cc:503] Master 127.12.8.254:33683 was elected leader, sending a full tablet report...
I20260812 02:33:09.306770 30611 raft_consensus.cc:348] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.307331 30608 ts_tablet_manager.cc:1428] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260812 02:33:09.307396 30611 raft_consensus.cc:374] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:09.307490 30611 raft_consensus.cc:729] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10a089ad5bff44fbb7812fa08360a752, State: Initialized, Role: FOLLOWER
I20260812 02:33:09.308461 30611 consensus_queue.cc:260] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.310135 30430 heartbeater.cc:503] Master 127.12.8.254:33683 was elected leader, sending a full tablet report...
I20260812 02:33:09.310688 30611 ts_tablet_manager.cc:1428] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260812 02:33:09.310700 30609 ts_tablet_manager.cc:1428] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
W20260812 02:33:09.370571 30262 tablet.cc:2369] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:33:09.415853 30591 tablet.cc:2369] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:09.418583 30617 raft_consensus.cc:482] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:33:09.418979 30617 raft_consensus.cc:504] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.420940 30617 leader_election.cc:288] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 250bddab426b4b6a911412ad5eeef234 (127.12.8.193:46521), 10a089ad5bff44fbb7812fa08360a752 (127.12.8.194:39459)
W20260812 02:33:09.424875 30431 tablet.cc:2369] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:09.427816 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.421658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:43157 (local address 127.12.8.193:46521)
0812 02:33:09.421997 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.422004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.422038 (+ 34us) server_negotiation.cc:407] Connection header received
0812 02:33:09.422148 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.422153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.422257 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.422393 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:09.422961 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.423714 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.425746 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.426765 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.426945 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.427034 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.427366 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.427521 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260812 02:33:09.427830 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.421404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:46521 (local address 127.12.8.195:43157)
0812 02:33:09.421727 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:09.421755 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:09.421972 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:09.422593 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:09.422597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:09.422628 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:09.422801 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.422809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.423863 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.423867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.425498 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.425513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.426958 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.426964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.427192 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.427222 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:33:09.427320 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 02:33:09.427488 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 02:33:09.428646 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.422086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:59061 (local address 127.12.8.194:39459)
0812 02:33:09.422313 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.422320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.422831 (+ 511us) server_negotiation.cc:407] Connection header received
0812 02:33:09.423026 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.423030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.423097 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.423177 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:09.423763 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.424568 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.426772 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.427628 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.427785 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.427863 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.428225 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.428365 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:33:09.428710 30619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.421838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:39459 (local address 127.12.8.195:59061)
0812 02:33:09.422714 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:09.422729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:09.422876 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:09.423216 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:09.423220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:09.423246 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:09.423610 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.423620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.424759 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.424764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.426563 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:09.426576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.427847 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.427852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:09.428024 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.428058 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 02:33:09.428191 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 02:33:09.428338 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":534,"threads_started":1}
I20260812 02:33:09.429126 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fad0d9aaed3499ab711a1209a2f6e05" candidate_uuid: "a952991dde9d4d96b6f04a389eca4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250bddab426b4b6a911412ad5eeef234" is_pre_election: true
I20260812 02:33:09.429950 30210 raft_consensus.cc:2454] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a952991dde9d4d96b6f04a389eca4035 in term 0.
I20260812 02:33:09.429900 30368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fad0d9aaed3499ab711a1209a2f6e05" candidate_uuid: "a952991dde9d4d96b6f04a389eca4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a089ad5bff44fbb7812fa08360a752" is_pre_election: true
I20260812 02:33:09.430455 30368 raft_consensus.cc:2454] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a952991dde9d4d96b6f04a389eca4035 in term 0.
I20260812 02:33:09.430941 30452 leader_election.cc:302] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250bddab426b4b6a911412ad5eeef234, a952991dde9d4d96b6f04a389eca4035; no voters:
I20260812 02:33:09.431399 30617 raft_consensus.cc:2781] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:33:09.431558 30617 raft_consensus.cc:482] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:33:09.431674 30617 raft_consensus.cc:3037] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:09.434015 30617 raft_consensus.cc:504] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.434657 30617 leader_election.cc:288] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [CANDIDATE]: Term 1 election: Requested vote from peers 250bddab426b4b6a911412ad5eeef234 (127.12.8.193:46521), 10a089ad5bff44fbb7812fa08360a752 (127.12.8.194:39459)
I20260812 02:33:09.435386 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fad0d9aaed3499ab711a1209a2f6e05" candidate_uuid: "a952991dde9d4d96b6f04a389eca4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "250bddab426b4b6a911412ad5eeef234"
I20260812 02:33:09.435359 30368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fad0d9aaed3499ab711a1209a2f6e05" candidate_uuid: "a952991dde9d4d96b6f04a389eca4035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10a089ad5bff44fbb7812fa08360a752"
I20260812 02:33:09.435636 30210 raft_consensus.cc:3037] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:09.435636 30368 raft_consensus.cc:3037] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:09.439261 30368 raft_consensus.cc:2454] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a952991dde9d4d96b6f04a389eca4035 in term 1.
I20260812 02:33:09.439318 30210 raft_consensus.cc:2454] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a952991dde9d4d96b6f04a389eca4035 in term 1.
I20260812 02:33:09.439829 30452 leader_election.cc:302] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 250bddab426b4b6a911412ad5eeef234, a952991dde9d4d96b6f04a389eca4035; no voters:
I20260812 02:33:09.440151 30617 raft_consensus.cc:2781] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:09.440651 30617 raft_consensus.cc:686] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 1 LEADER]: Becoming Leader. State: Replica: a952991dde9d4d96b6f04a389eca4035, State: Running, Role: LEADER
I20260812 02:33:09.441248 30617 consensus_queue.cc:237] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } }
I20260812 02:33:09.446103 30042 catalog_manager.cc:5495] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 reported cstate change: term changed from 0 to 1, leader changed from <none> to a952991dde9d4d96b6f04a389eca4035 (127.12.8.195). New cstate: current_term: 1 leader_uuid: "a952991dde9d4d96b6f04a389eca4035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a952991dde9d4d96b6f04a389eca4035" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 45791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 } health_report { overall_health: UNKNOWN } } }
I20260812 02:33:09.490420 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.485083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.12.8.195:45791)
0812 02:33:09.485380 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.485387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.485407 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:33:09.485499 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.485502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.485584 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.485713 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:33:09.486122 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.486837 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.488314 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.488708 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.488809 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.489321 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:33:09.489818 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:33:09.489959 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.490063 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.490146 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 02:33:09.500823 30368 raft_consensus.cc:1261] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 1 FOLLOWER]: Refusing update from remote peer a952991dde9d4d96b6f04a389eca4035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:09.501072 30210 raft_consensus.cc:1261] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 1 FOLLOWER]: Refusing update from remote peer a952991dde9d4d96b6f04a389eca4035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:09.502130 30620 consensus_queue.cc:1035] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10a089ad5bff44fbb7812fa08360a752" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:09.502625 30617 consensus_queue.cc:1035] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "250bddab426b4b6a911412ad5eeef234" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:09.540639 30041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:09.541134 30041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:09.545614 30041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Sending DeleteTablet for 3 replicas of tablet 8fad0d9aaed3499ab711a1209a2f6e05
I20260812 02:33:09.547071 30183 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fad0d9aaed3499ab711a1209a2f6e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:09 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:47678
I20260812 02:33:09.547112 30510 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fad0d9aaed3499ab711a1209a2f6e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:09 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:35912
I20260812 02:33:09.547302 30346 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fad0d9aaed3499ab711a1209a2f6e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:09 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:33720
I20260812 02:33:09.547389 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 30087
I20260812 02:33:09.548501 30638 tablet_replica.cc:330] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: stopping tablet replica
I20260812 02:33:09.549023 30638 raft_consensus.cc:2229] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:33:09.549809 30639 tablet_replica.cc:330] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: stopping tablet replica
I20260812 02:33:09.550287 30639 raft_consensus.cc:2229] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:09.550889 30639 raft_consensus.cc:2258] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:09.551708 30637 tablet_replica.cc:330] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: stopping tablet replica
I20260812 02:33:09.552150 30637 raft_consensus.cc:2229] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:09.552762 30637 raft_consensus.cc:2258] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:09.553944 30638 raft_consensus.cc:2258] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:09.555305 30639 ts_tablet_manager.cc:1905] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:09.556206 30637 ts_tablet_manager.cc:1905] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:09.564059 30638 ts_tablet_manager.cc:1905] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:09.564997 30637 ts_tablet_manager.cc:1918] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:09.565333 30637 log.cc:1196] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/8fad0d9aaed3499ab711a1209a2f6e05
I20260812 02:33:09.566057 30637 ts_tablet_manager.cc:1939] T 8fad0d9aaed3499ab711a1209a2f6e05 P 250bddab426b4b6a911412ad5eeef234: Deleting consensus metadata
I20260812 02:33:09.568691 30025 catalog_manager.cc:4841] TS 250bddab426b4b6a911412ad5eeef234 (127.12.8.193:46521): tablet 8fad0d9aaed3499ab711a1209a2f6e05 (table test-table [id=70b416a44ede41f2a8ba501c53bb72e5]) successfully deleted
I20260812 02:33:09.569393 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.548421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.12.8.193:46521)
0812 02:33:09.549885 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.549892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.549918 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:33:09.549996 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.550001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.550134 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.550295 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:09.552242 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.553518 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.556008 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.556754 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.556930 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.560665 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:09.560683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:09.560707 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:09.560766 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:09.565362 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.566370 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.566382 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.566386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.566682 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.567162 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.567167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.567170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.567837 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:09.568188 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.568688 (+ 500us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.568955 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:33:09.570116 30254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:09.571142 30639 ts_tablet_manager.cc:1918] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:09.571372 30639 log.cc:1196] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/8fad0d9aaed3499ab711a1209a2f6e05
I20260812 02:33:09.571978 30639 ts_tablet_manager.cc:1939] T 8fad0d9aaed3499ab711a1209a2f6e05 P 10a089ad5bff44fbb7812fa08360a752: Deleting consensus metadata
I20260812 02:33:09.574061 30025 catalog_manager.cc:4841] TS 10a089ad5bff44fbb7812fa08360a752 (127.12.8.194:39459): tablet 8fad0d9aaed3499ab711a1209a2f6e05 (table test-table [id=70b416a44ede41f2a8ba501c53bb72e5]) successfully deleted
I20260812 02:33:09.575049 30638 ts_tablet_manager.cc:1918] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:09.575273 30638 log.cc:1196] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/8fad0d9aaed3499ab711a1209a2f6e05
I20260812 02:33:09.575964 30638 ts_tablet_manager.cc:1939] T 8fad0d9aaed3499ab711a1209a2f6e05 P a952991dde9d4d96b6f04a389eca4035: Deleting consensus metadata
I20260812 02:33:09.578627 30028 catalog_manager.cc:4841] TS a952991dde9d4d96b6f04a389eca4035 (127.12.8.195:45791): tablet 8fad0d9aaed3499ab711a1209a2f6e05 (table test-table [id=70b416a44ede41f2a8ba501c53bb72e5]) successfully deleted
I20260812 02:33:09.674211 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 30087
I20260812 02:33:09.707827 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 30265
I20260812 02:33:09.723649 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.708952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.12.8.194:39459)
0812 02:33:09.709244 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.709251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.709282 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:33:09.709348 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.709353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.709446 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.709557 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:09.710977 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.711775 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.713164 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.713706 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.713829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.716951 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:09.716970 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:09.716989 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:09.717030 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:09.720460 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.721332 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.721340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.721343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.721489 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.721905 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.721908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.721910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.722350 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:09.722652 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.723014 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.723255 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 02:33:09.724829 30424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:09.827752 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 30265
I20260812 02:33:09.851665 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 30434
I20260812 02:33:09.868571 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:09.852671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.12.8.195:45791)
0812 02:33:09.852880 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:09.852886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:09.853057 (+ 171us) server_negotiation.cc:407] Connection header received
0812 02:33:09.853273 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:09.853277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:09.853345 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:09.853443 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:09.855038 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.855921 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.857457 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:09.857932 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:09.858061 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:09.861133 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:09.861151 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:09.861167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:09.861326 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:09.865110 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.865999 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.866008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.866010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.866213 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:09.866625 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:09.866629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:09.866631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:09.867070 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:09.867622 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:09.867853 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 02:33:09.868101 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 02:33:09.869633 30585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:10.008951 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 30434
I20260812 02:33:10.034991 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 30005
I20260812 02:33:10.036310 30079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:10.172465 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 30005
I20260812 02:33:10.206688 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33683
--webserver_interface=127.12.8.254
--webserver_port=34589
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:33683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:10.569157 30646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:10.569535 30646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:10.569630 30646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:10.577694 30646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:33:10.577850 30646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:10.577900 30646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:10.577971 30646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:33:10.578020 30646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:33:10.588721 30646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:33683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=34589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:10.590328 30646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:10.591750 30646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:10.605692 30658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:10.605701 30655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:10.605763 30656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:10.607581 30646 server_base.cc:1034] running on GCE node
I20260812 02:33:10.608616 30646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:10.610508 30646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:10.611835 30646 hybrid_clock.cc:648] HybridClock initialized: now 1786501990611732 us; error 64 us; skew 500 ppm
I20260812 02:33:10.612330 30646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:10 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501990, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:10.616017 30646 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:33:10.618002 30646 webserver.cc:460] Webserver started at http://127.12.8.254:34589/ using document root <none> and password file <none>
I20260812 02:33:10.618777 30646 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:10.618896 30646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:10.625749 30646 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260812 02:33:10.632202 30666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:10.634320 30646 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 02:33:10.634558 30646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "56d2cb297c55473a9a13ba547e3e22a7"
format_stamp: "Formatted at 2026-08-12 02:33:03 on dist-test-slave-vq2q"
server_key: "0d46e4883bba37b24a2be0c27c650200"
server_key_iv: "825e4b5ab43da26d6adbc2b2887533bd"
server_key_version: "encryptionkey@0"
I20260812 02:33:10.635710 30646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:10.666903 30646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:10.671475 30646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:10.671765 30646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:10.694614 30646 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:33683
I20260812 02:33:10.694597 30727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:33683 every 8 connection(s)
I20260812 02:33:10.696413 30646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:33:10.703138 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 30646
I20260812 02:33:10.704437 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:46521
--local_ip_for_outbound_sockets=127.12.8.193
--tserver_master_addrs=127.12.8.254:33683
--webserver_port=36519
--webserver_interface=127.12.8.193
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:33:10.707542 30728 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:33:10.713737 30728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Bootstrap starting.
I20260812 02:33:10.743520 30728 log.cc:824] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:10.767318 30728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:33:10.768087 30728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7: Bootstrap complete.
I20260812 02:33:10.780802 30728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:10.782146 30728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d2cb297c55473a9a13ba547e3e22a7, State: Initialized, Role: FOLLOWER
I20260812 02:33:10.783099 30728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:10.783413 30728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:33:10.783690 30728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:33:10.784017 30728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:33:10.790832 30728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:10.791726 30728 leader_election.cc:302] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d2cb297c55473a9a13ba547e3e22a7; no voters:
I20260812 02:33:10.792717 30728 leader_election.cc:288] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 02:33:10.792932 30732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:33:10.794543 30732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [term 3 LEADER]: Becoming Leader. State: Replica: 56d2cb297c55473a9a13ba547e3e22a7, State: Running, Role: LEADER
I20260812 02:33:10.795430 30732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } }
I20260812 02:33:10.796144 30728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:33:10.801247 30735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d2cb297c55473a9a13ba547e3e22a7. Latest consensus state: current_term: 3 leader_uuid: "56d2cb297c55473a9a13ba547e3e22a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } } }
I20260812 02:33:10.801832 30735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:10.801155 30734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "56d2cb297c55473a9a13ba547e3e22a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d2cb297c55473a9a13ba547e3e22a7" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33683 } } }
I20260812 02:33:10.802452 30734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d2cb297c55473a9a13ba547e3e22a7 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:10.804323 30740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:33:10.818179 30740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:33:10.821324 30740 catalog_manager.cc:1259] Loaded cluster ID: 56f9dbc527224550822c1bbda21152a9
I20260812 02:33:10.821681 30740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 02:33:10.825230 30740 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa47c5799ad google::LogMessage::Fail() at ??:0
@ 0x7fa47c57c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa47c5794de google::LogMessage::Flush() at ??:0
@ 0x7fa47c57aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa491fac512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa491fa9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa492031610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa492031439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa47f1c5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa47f1c891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa47f1c8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa47f1a3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa48247d6db start_thread at ??:0
@ 0x7fa47b41361f clone at ??:0
W20260812 02:33:11.147445 30730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:11.147755 30730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:11.147891 30730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:11.155369 30730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:11.155515 30730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:11.155675 30730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:33:11.165956 30730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:46521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=36519
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:11.167385 30730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:11.168757 30730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:11.183660 30774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:11.183908 30775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:11.184304 30777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:11.185056 30730 server_base.cc:1034] running on GCE node
I20260812 02:33:11.185951 30730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:11.187304 30730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:11.188560 30730 hybrid_clock.cc:648] HybridClock initialized: now 1786501991188495 us; error 40 us; skew 500 ppm
I20260812 02:33:11.188998 30730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:11 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501991, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:11.193138 30730 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:33:11.195143 30730 webserver.cc:460] Webserver started at http://127.12.8.193:36519/ using document root <none> and password file <none>
I20260812 02:33:11.196050 30730 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:11.196197 30730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:11.203395 30730 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 02:33:11.207255 30784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:11.209048 30730 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:33:11.209318 30730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "250bddab426b4b6a911412ad5eeef234"
format_stamp: "Formatted at 2026-08-12 02:33:04 on dist-test-slave-vq2q"
server_key: "b8db35c22b851794731a1ec0b4ba717a"
server_key_iv: "48fdba4529acfa3b58474761bcd097b5"
server_key_version: "encryptionkey@0"
I20260812 02:33:11.210446 30730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:11.226383 30730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:11.230823 30730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:11.231088 30730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:11.232154 30730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:11.233577 30730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:11.233673 30730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:11.233804 30730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:11.233881 30730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:11.267445 30730 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:46521
I20260812 02:33:11.267516 30921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:46521 every 8 connection(s)
I20260812 02:33:11.268955 30730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:33:11.273871 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:11.271969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:32979)
0812 02:33:11.272758 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:11.272959 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":325,"threads_started":1}
W20260812 02:33:11.274580 30923 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:33683 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.254:33683: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
I20260812 02:33:11.275518 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:11.275193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:36073)
0812 02:33:11.275332 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:11.275408 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 02:33:11.276262 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:11.275977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:43903)
0812 02:33:11.276085 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:11.276156 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 02:33:11.276494 30923 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:33:11.278424 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 30730
I20260812 02:33:11.279088 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:39459
--local_ip_for_outbound_sockets=127.12.8.194
--tserver_master_addrs=127.12.8.254:33683
--webserver_port=42019
--webserver_interface=127.12.8.194
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:11.627053 30928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:11.627333 30928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:11.627441 30928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:11.635279 30928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:11.635403 30928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:11.635524 30928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:33:11.645691 30928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:39459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=42019
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:11.647049 30928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:11.648331 30928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:11.661693 30933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:11.661716 30934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:11.661962 30936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:11.662479 30928 server_base.cc:1034] running on GCE node
I20260812 02:33:11.663082 30928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:11.664034 30928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:11.665271 30928 hybrid_clock.cc:648] HybridClock initialized: now 1786501991665215 us; error 43 us; skew 500 ppm
I20260812 02:33:11.665642 30928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:11 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501991, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:11.673238 30928 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:33:11.674793 30928 webserver.cc:460] Webserver started at http://127.12.8.194:42019/ using document root <none> and password file <none>
I20260812 02:33:11.675531 30928 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:11.675644 30928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:11.681411 30928 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:33:11.684309 30945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:11.685781 30928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:33:11.685941 30928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "10a089ad5bff44fbb7812fa08360a752"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "63ab828e07b5c6c3113d2c648dc1057e"
server_key_iv: "5ea897b244df734974182369d95ffa3d"
server_key_version: "encryptionkey@0"
I20260812 02:33:11.686956 30928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:11.711375 30928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:11.715757 30928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:11.715996 30928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:11.717007 30928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:11.718684 30928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:11.718798 30928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:11.718904 30928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:11.718984 30928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:11.750602 30928 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:39459
I20260812 02:33:11.750670 31081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:39459 every 8 connection(s)
I20260812 02:33:11.752156 30928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:33:11.756950 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:11.755084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:38967)
0812 02:33:11.755848 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:11.756084 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":291,"threads_started":1}
W20260812 02:33:11.757682 31082 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:33683 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.254:33683: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
I20260812 02:33:11.758607 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:11.758223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:47155)
0812 02:33:11.758402 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:11.758479 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 02:33:11.759462 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:11.759119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:59865)
0812 02:33:11.759257 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:11.759346 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 02:33:11.759704 31082 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:33:11.762339 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 30928
I20260812 02:33:11.762951 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:45791
--local_ip_for_outbound_sockets=127.12.8.195
--tserver_master_addrs=127.12.8.254:33683
--webserver_port=34961
--webserver_interface=127.12.8.195
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:12.110482 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:12.110793 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:12.110929 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:12.118327 31086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:12.118465 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:12.118621 31086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:33:12.128672 31086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:45791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=34961
--tserver_master_addrs=127.12.8.254:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:12.130112 31086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:12.131536 31086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:12.144588 31095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:12.144615 31093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:12.144810 31099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:12.145538 31086 server_base.cc:1034] running on GCE node
I20260812 02:33:12.146155 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:12.147099 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:12.148313 31086 hybrid_clock.cc:648] HybridClock initialized: now 1786501992148255 us; error 50 us; skew 500 ppm
I20260812 02:33:12.148633 31086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:12 dist-test-slave-vq2q krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:12.151821 31086 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:33:12.153344 31086 webserver.cc:460] Webserver started at http://127.12.8.195:34961/ using document root <none> and password file <none>
I20260812 02:33:12.154023 31086 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:12.154143 31086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:12.160807 31086 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 02:33:12.163633 31107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:12.164989 31086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 02:33:12.165150 31086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "a952991dde9d4d96b6f04a389eca4035"
format_stamp: "Formatted at 2026-08-12 02:33:05 on dist-test-slave-vq2q"
server_key: "ebb6d9112d1d0f7e4400a3629d84722f"
server_key_iv: "8de5d9dbb4000e6c481f847509dd1e06"
server_key_version: "encryptionkey@0"
I20260812 02:33:12.166287 31086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:12.183053 31086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:12.187114 31086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:12.187393 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:12.188354 31086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:12.189747 31086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:12.189860 31086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:12.189966 31086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:12.190042 31086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:12.222361 31086 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:45791
I20260812 02:33:12.222455 31244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:45791 every 8 connection(s)
I20260812 02:33:12.223882 31086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:33:12.226169 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 31086
I20260812 02:33:12.229408 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:12.227142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:34347)
0812 02:33:12.228049 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:12.228351 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":341,"threads_started":1}
W20260812 02:33:12.230084 31246 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:33683 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.254:33683: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
I20260812 02:33:12.230836 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:12.230523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:52431)
0812 02:33:12.230665 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:12.230733 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 02:33:12.231650 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:12.231331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:56135)
0812 02:33:12.231466 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:12.231546 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 02:33:12.231879 31246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:33:12.278288 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:12.277118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:50423)
0812 02:33:12.277827 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:12.277988 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":339,"threads_started":1}
I20260812 02:33:12.762251 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:12.760963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:42111)
0812 02:33:12.761804 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:12.761955 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":416,"threads_started":1}
I20260812 02:33:13.233897 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:13.232686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:42467)
0812 02:33:13.233386 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:13.233559 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":299,"threads_started":1}
I20260812 02:33:13.280864 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:13.279728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:60945)
0812 02:33:13.280498 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:13.280625 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":369,"threads_started":1}
I20260812 02:33:13.764569 31259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:13.763511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:40155)
0812 02:33:13.764191 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:13.764338 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":302,"threads_started":1}
I20260812 02:33:14.236392 31260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:14.235268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:45599)
0812 02:33:14.235941 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:14.236107 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":327,"threads_started":1}
I20260812 02:33:14.283325 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:14.282219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:39787)
0812 02:33:14.282944 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:14.283059 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":366,"threads_started":1}
I20260812 02:33:14.766973 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:14.765850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:33221)
0812 02:33:14.766583 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:14.766720 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":369,"threads_started":1}
I20260812 02:33:15.238914 31264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:15.237786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:51619)
0812 02:33:15.238534 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:15.238649 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":440,"threads_started":1}
I20260812 02:33:15.285692 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:15.284664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:60219)
0812 02:33:15.285313 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:15.285414 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":323,"threads_started":1}
I20260812 02:33:15.769317 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:15.768232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:52827)
0812 02:33:15.768878 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:15.768990 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":328,"threads_started":1}
I20260812 02:33:16.241304 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:16.240245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:39931)
0812 02:33:16.240823 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:16.240931 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":312,"threads_started":1}
I20260812 02:33:16.288182 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:16.287186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:58463)
0812 02:33:16.287869 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:16.287960 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":389,"threads_started":1}
I20260812 02:33:16.771706 31272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:16.770681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:34687)
0812 02:33:16.771390 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:16.771491 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":410,"threads_started":1}
I20260812 02:33:17.243539 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:17.242563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:49427)
0812 02:33:17.243218 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:17.243323 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":342,"threads_started":1}
I20260812 02:33:17.290486 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:17.289443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:44517)
0812 02:33:17.290078 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:17.290238 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":304,"threads_started":1}
I20260812 02:33:17.774328 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:17.773075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:33689)
0812 02:33:17.773892 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:17.774066 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":403,"threads_started":1}
I20260812 02:33:18.245819 31276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:18.244621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:50363)
0812 02:33:18.245445 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:18.245582 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":415,"threads_started":1}
I20260812 02:33:18.292950 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:18.291798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:39951)
0812 02:33:18.292556 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:18.292678 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":290,"threads_started":1}
I20260812 02:33:18.776849 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:18.775698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:33163)
0812 02:33:18.776464 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:18.776578 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":314,"threads_started":1}
I20260812 02:33:19.248170 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:19.247009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:55923)
0812 02:33:19.247789 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:19.247909 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":317,"threads_started":1}
I20260812 02:33:19.295303 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:19.294285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:36445)
0812 02:33:19.294958 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:19.295081 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":332,"threads_started":1}
I20260812 02:33:19.778990 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:19.778106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:59643)
0812 02:33:19.778699 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:19.778803 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":328,"threads_started":1}
I20260812 02:33:20.250315 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:20.249407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:52451)
0812 02:33:20.250008 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:20.250120 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":282,"threads_started":1}
I20260812 02:33:20.297482 31284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:20.296323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:44027)
0812 02:33:20.297064 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:20.297241 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":366,"threads_started":1}
I20260812 02:33:20.781296 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:20.780263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:59361)
0812 02:33:20.780916 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:20.781015 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":314,"threads_started":1}
I20260812 02:33:21.252830 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:21.251689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:55903)
0812 02:33:21.252472 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:21.252596 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":362,"threads_started":1}
I20260812 02:33:21.299724 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:21.298789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:55993)
0812 02:33:21.299416 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:21.299531 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":328,"threads_started":1}
I20260812 02:33:21.783367 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:21.782472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:47955)
0812 02:33:21.783044 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:21.783153 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":306,"threads_started":1}
I20260812 02:33:22.255327 31290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:22.254405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:43093)
0812 02:33:22.255004 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:22.255108 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":343,"threads_started":1}
I20260812 02:33:22.301759 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:22.300820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:44045)
0812 02:33:22.301424 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:22.301521 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":351,"threads_started":1}
I20260812 02:33:22.785821 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:22.784732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:34113)
0812 02:33:22.785419 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:22.785567 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":326,"threads_started":1}
I20260812 02:33:23.257555 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:23.256601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:57809)
0812 02:33:23.257222 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:23.257348 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":297,"threads_started":1}
I20260812 02:33:23.303957 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:23.302970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:47505)
0812 02:33:23.303624 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:23.303747 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":343,"threads_started":1}
I20260812 02:33:23.787976 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:23.787106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:48875)
0812 02:33:23.787665 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:23.787783 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":272,"threads_started":1}
I20260812 02:33:24.260059 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:24.259078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:40875)
0812 02:33:24.259722 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:24.259860 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":321,"threads_started":1}
I20260812 02:33:24.306492 31302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:24.305417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:33985)
0812 02:33:24.306125 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:24.306258 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":390,"threads_started":1}
I20260812 02:33:24.790238 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:24.789024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:52705)
0812 02:33:24.789770 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:24.789925 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":358,"threads_started":1}
I20260812 02:33:25.262560 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:25.261345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:45949)
0812 02:33:25.262158 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:25.262303 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":388,"threads_started":1}
I20260812 02:33:25.308635 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:25.307681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:57101)
0812 02:33:25.308308 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:25.308421 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":303,"threads_started":1}
I20260812 02:33:25.792426 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:25.791405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:59201)
0812 02:33:25.792078 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:25.792188 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":367,"threads_started":1}
I20260812 02:33:26.264760 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:26.263697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:46315)
0812 02:33:26.264426 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:26.264521 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":344,"threads_started":1}
I20260812 02:33:26.311058 31309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:26.310104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.193:35119)
0812 02:33:26.310706 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:26.310818 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":321,"threads_started":1}
I20260812 02:33:26.794533 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:26.793685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.194:35063)
0812 02:33:26.794188 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:26.794314 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":266,"threads_started":1}
I20260812 02:33:27.236120 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 30730
I20260812 02:33:27.254911 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:27.237314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.12.8.193:46521)
0812 02:33:27.238156 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:27.238171 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:27.238211 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:33:27.239958 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:27.239967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:27.240396 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:27.240678 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:27.242275 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:27.243513 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:27.244759 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:27.245530 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:27.245691 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:27.248257 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:27.248265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:27.248280 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:27.248321 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:27.251261 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:27.252092 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:27.252099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:27.252106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:27.252296 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:27.252768 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:27.252772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:27.252774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:27.253433 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:27.253790 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:27.254213 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 02:33:27.254592 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":511,"threads_started":1}
I20260812 02:33:27.256325 30916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:27.266932 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:27.266072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:33683 (local address 127.12.8.195:57307)
0812 02:33:27.266637 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:27.266737 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:33683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":298,"threads_started":1}
I20260812 02:33:27.341588 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 30730
I20260812 02:33:27.362006 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 30928
I20260812 02:33:27.379820 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:27.363028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.12.8.194:39459)
0812 02:33:27.363597 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:27.363612 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:27.363649 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:33:27.365290 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:27.365301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:27.365738 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:27.366032 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:27.367403 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:27.368640 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:27.370054 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:27.370821 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:27.370960 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:27.373594 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:27.373602 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:27.373617 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:27.373660 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:27.376406 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:27.377233 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:27.377245 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:27.377253 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:27.377454 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:27.377800 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:27.377803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:27.377805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:27.378352 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:27.378600 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:27.379091 (+ 491us) server_negotiation.cc:299] Negotiation successful
0812 02:33:27.379500 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":254,"threads_started":1}
I20260812 02:33:27.381307 31076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:27.465348 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 30928
I20260812 02:33:27.487178 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 31086
I20260812 02:33:27.504565 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:27.488091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.12.8.195:45791)
0812 02:33:27.488689 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:27.488703 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:27.488790 (+ 87us) server_negotiation.cc:407] Connection header received
0812 02:33:27.490832 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:27.490842 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:27.491275 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:27.491581 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:27.492982 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:27.494257 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:27.495450 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:27.496091 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:27.496219 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:27.498808 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:27.498815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:27.498829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:27.498871 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:27.501727 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:27.502359 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:27.502364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:27.502370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:27.502552 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:27.502845 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:27.502848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:27.502850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:27.503345 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:27.503600 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:27.503928 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 02:33:27.504239 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":271,"threads_started":1}
I20260812 02:33:27.506000 31239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:27.589006 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 31086
2026-08-12T02:33:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25798 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:33:27 dist-test-slave-vq2q krb5kdc[31324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:33:27 dist-test-slave-vq2q krb5kdc[31324](info): set up 2 sockets
Aug 12 02:33:27 dist-test-slave-vq2q krb5kdc[31324](info): commencing operation
krb5kdc: starting...
W20260812 02:33:28.752853 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:33:29 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:33:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:33:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:33:29.157824 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43849
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:43849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:29.586118 31337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:29.586527 31337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:29.586658 31337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:29.594993 31337 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:33:29.595147 31337 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:33:29.595191 31337 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:33:29.595243 31337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:33:29.595304 31337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:29.595383 31337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:29.595432 31337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:33:29.595490 31337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:33:29.606963 31337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:43849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:29.608711 31337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:29.610467 31337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:29.624516 31344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:29.624621 31348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:29.624529 31345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:29.625514 31337 server_base.cc:1034] running on GCE node
I20260812 02:33:29.626533 31337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:29.628366 31337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:29.629613 31337 hybrid_clock.cc:648] HybridClock initialized: now 1786502009629542 us; error 45 us; skew 500 ppm
I20260812 02:33:29.630038 31337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:29 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502009, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:29.633812 31337 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:33:29.635661 31337 webserver.cc:460] Webserver started at http://127.12.8.254:41331/ using document root <none> and password file <none>
I20260812 02:33:29.636533 31337 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:29.636633 31337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:29.636973 31337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:29.641168 31337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "141e89072b794cfb95c510528e572f6e"
format_stamp: "Formatted at 2026-08-12 02:33:29 on dist-test-slave-vq2q"
server_key: "0209c0bc22897e0df9f31200d8a8d174"
server_key_iv: "bfe22878fbebd96fe6118354d95b9f60"
server_key_version: "encryptionkey@0"
I20260812 02:33:29.642185 31337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "141e89072b794cfb95c510528e572f6e"
format_stamp: "Formatted at 2026-08-12 02:33:29 on dist-test-slave-vq2q"
server_key: "0209c0bc22897e0df9f31200d8a8d174"
server_key_iv: "bfe22878fbebd96fe6118354d95b9f60"
server_key_version: "encryptionkey@0"
I20260812 02:33:29.648072 31337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:33:29.651939 31356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:29.653469 31337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:33:29.653632 31337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "141e89072b794cfb95c510528e572f6e"
format_stamp: "Formatted at 2026-08-12 02:33:29 on dist-test-slave-vq2q"
server_key: "0209c0bc22897e0df9f31200d8a8d174"
server_key_iv: "bfe22878fbebd96fe6118354d95b9f60"
server_key_version: "encryptionkey@0"
I20260812 02:33:29.653893 31337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:29.677109 31337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:29.681734 31337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:29.682003 31337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:29.700121 31337 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:43849
I20260812 02:33:29.700165 31417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:43849 every 8 connection(s)
I20260812 02:33:29.701570 31337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:33:29.705544 31418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:29.710907 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 31337
I20260812 02:33:29.711146 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:33:29.711952 12323 external_mini_cluster.cc:1428] Setting key 2823ea9608a35427d3d9382af282fb5e
I20260812 02:33:29.717293 31418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Bootstrap starting.
I20260812 02:33:29.720966 31418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:29.722286 31418 log.cc:824] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:29.725883 31418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: No bootstrap required, opened a new log
Aug 12 02:33:29 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:29.735975 31418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:33:29.736497 31418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:29.736627 31418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141e89072b794cfb95c510528e572f6e, State: Initialized, Role: FOLLOWER
I20260812 02:33:29.737476 31418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:33:29.737737 31418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:33:29.737888 31418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:33:29.738047 31418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:29.738257 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:29.714197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.12.8.254:43849)
0812 02:33:29.715076 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:29.715088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:29.715130 (+ 42us) server_negotiation.cc:407] Connection header received
0812 02:33:29.716801 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:29.716824 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:29.717257 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:29.717605 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:29.719529 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:29.720802 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:29.722775 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:29.723867 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:29.723998 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:29.729245 (+ 5247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:29.729257 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:29.729279 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:29.729337 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:29.733586 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:29.734414 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:29.734424 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:29.734434 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:29.734668 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:29.735057 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:29.735063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:29.735066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:29.735846 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:29.736154 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:29.736543 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 02:33:29.737046 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":419,"threads_started":1}
I20260812 02:33:29.740962 31418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:33:29.741568 31418 leader_election.cc:302] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 141e89072b794cfb95c510528e572f6e; no voters:
I20260812 02:33:29.742148 31418 leader_election.cc:288] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:33:29.742271 31425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:29.743481 31425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 1 LEADER]: Becoming Leader. State: Replica: 141e89072b794cfb95c510528e572f6e, State: Running, Role: LEADER
I20260812 02:33:29.744076 31425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:33:29.744437 31418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:33:29.746724 31426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "141e89072b794cfb95c510528e572f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } } }
I20260812 02:33:29.747103 31426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:29.747553 31427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 141e89072b794cfb95c510528e572f6e. Latest consensus state: current_term: 1 leader_uuid: "141e89072b794cfb95c510528e572f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } } }
I20260812 02:33:29.747903 31427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:29.750706 31433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:33:29.755471 31433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:33:29.764605 31433 catalog_manager.cc:1347] Generated new cluster ID: ebe5bee99f51431183725a24be9a3f4e
I20260812 02:33:29.764750 31433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:33:29.785097 31433 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:33:29.786787 31433 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:33:29.795095 31433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Generated new TSK 0
I20260812 02:33:29.796370 31433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:33:29.878826 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43849
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:30.252072 31449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:30.252367 31449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:30.252504 31449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:30.259945 31449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:30.260074 31449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:30.260228 31449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:33:30.270156 31449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:30.271481 31449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:30.272809 31449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:30.285773 31460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:30.285898 31457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:30.285876 31458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:30.287042 31449 server_base.cc:1034] running on GCE node
I20260812 02:33:30.287662 31449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:30.288635 31449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:30.289846 31449 hybrid_clock.cc:648] HybridClock initialized: now 1786502010289782 us; error 43 us; skew 500 ppm
I20260812 02:33:30.290153 31449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:30 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502010, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:30.293495 31449 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:33:30.295185 31449 webserver.cc:460] Webserver started at http://127.12.8.193:37613/ using document root <none> and password file <none>
I20260812 02:33:30.295902 31449 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:30.296026 31449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:30.296336 31449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:30.299468 31449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "b7ebadf893584046a0e6b815c049d5d4"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "41a6cddc0b539c48fe4635f9e55b9e78"
server_key_iv: "3c7083d9ce4d99339e712e83abdb6f23"
server_key_version: "encryptionkey@0"
I20260812 02:33:30.300244 31449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "b7ebadf893584046a0e6b815c049d5d4"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "41a6cddc0b539c48fe4635f9e55b9e78"
server_key_iv: "3c7083d9ce4d99339e712e83abdb6f23"
server_key_version: "encryptionkey@0"
I20260812 02:33:30.305552 31449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 02:33:30.309032 31467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:30.310448 31449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:33:30.310621 31449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "b7ebadf893584046a0e6b815c049d5d4"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "41a6cddc0b539c48fe4635f9e55b9e78"
server_key_iv: "3c7083d9ce4d99339e712e83abdb6f23"
server_key_version: "encryptionkey@0"
I20260812 02:33:30.310824 31449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:30.335438 31449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:30.339726 31449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:30.340018 31449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:30.341042 31449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:30.342554 31449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:30.342634 31449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:30.342756 31449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:30.342811 31449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:30.375161 31449 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:38837
I20260812 02:33:30.375217 31604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:38837 every 8 connection(s)
I20260812 02:33:30.376675 31449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:33:30.382354 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 31449
I20260812 02:33:30.382563 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:33:30.383203 12323 external_mini_cluster.cc:1428] Setting key 6b8ce7f62179b662d46c1fd3cf71b452
Aug 12 02:33:30 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502010, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:30.397327 31608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:30.379501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:46815 (local address 127.12.8.254:43849)
0812 02:33:30.379986 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:30.379991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:30.380926 (+ 935us) server_negotiation.cc:407] Connection header received
0812 02:33:30.382913 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:30.382919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:30.383003 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:30.383099 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:30.384799 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:30.385677 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:30.387274 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:30.387686 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:30.387811 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:30.391613 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:30.391617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:30.391621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:30.391657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:30.394178 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:30.394928 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:30.394932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:30.394935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:30.395060 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:30.395494 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:30.395498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:30.395500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:30.395776 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:30.395917 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:30.396674 (+ 757us) server_negotiation.cc:299] Negotiation successful
0812 02:33:30.396918 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260812 02:33:30.398306 31609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:30.380002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:46815)
0812 02:33:30.380749 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:30.380797 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:30.382621 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:30.383361 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:30.383374 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:30.383921 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:30.384624 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:30.384643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:30.385864 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:30.385868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:30.387120 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:30.387133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:30.387934 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:30.387941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:30.388182 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:30.388909 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:30.388935 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:30.391377 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:30.394327 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:30.394337 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:30.394355 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:30.394805 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:30.395169 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:30.395175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:30.395178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:30.395382 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:30.395907 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:30.395915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:30.396277 (+ 362us) client_negotiation.cc:769] Sending connection context
0812 02:33:30.396539 (+ 262us) client_negotiation.cc:240] Negotiation successful
0812 02:33:30.397124 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":296,"threads_started":1}
I20260812 02:33:30.400027 31606 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43849
I20260812 02:33:30.400395 31606 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:30.401294 31606 heartbeater.cc:511] Master 127.12.8.254:43849 requested a full tablet report, sending...
I20260812 02:33:30.403738 31374 ts_manager.cc:194] Registered new tserver with Master: b7ebadf893584046a0e6b815c049d5d4 (127.12.8.193:38837)
I20260812 02:33:30.405792 31374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:46815
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:33:30.452894 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43849
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:30.795492 31613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:30.795773 31613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:30.795878 31613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:30.803210 31613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:30.803320 31613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:30.803442 31613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:33:30.813505 31613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:30.814877 31613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:30.816219 31613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:30.829732 31619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:30.829741 31620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:30.829659 31622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:30.830397 31613 server_base.cc:1034] running on GCE node
I20260812 02:33:30.831063 31613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:30.832108 31613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:30.833313 31613 hybrid_clock.cc:648] HybridClock initialized: now 1786502010833262 us; error 43 us; skew 500 ppm
I20260812 02:33:30.833630 31613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:30 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502010, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:30.836796 31613 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:33:30.838431 31613 webserver.cc:460] Webserver started at http://127.12.8.194:38889/ using document root <none> and password file <none>
I20260812 02:33:30.839116 31613 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:30.839236 31613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:30.839534 31613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:30.842554 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "03b245e6a7734141a86af164d77d5dc2"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "678661c7a9c5f4bb7c0ddb43923daf58"
server_key_iv: "5774faa88524a3570b25ac92650e0a63"
server_key_version: "encryptionkey@0"
I20260812 02:33:30.843279 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "03b245e6a7734141a86af164d77d5dc2"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "678661c7a9c5f4bb7c0ddb43923daf58"
server_key_iv: "5774faa88524a3570b25ac92650e0a63"
server_key_version: "encryptionkey@0"
I20260812 02:33:30.848439 31613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 02:33:30.852197 31631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:30.853581 31613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:33:30.853749 31613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "03b245e6a7734141a86af164d77d5dc2"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "678661c7a9c5f4bb7c0ddb43923daf58"
server_key_iv: "5774faa88524a3570b25ac92650e0a63"
server_key_version: "encryptionkey@0"
I20260812 02:33:30.854008 31613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:30.880342 31613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:30.884646 31613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:30.884935 31613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:30.885980 31613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:30.887316 31613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:30.887404 31613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:30.887527 31613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:30.887583 31613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:30.920150 31613 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:44233
I20260812 02:33:30.920202 31764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:44233 every 8 connection(s)
I20260812 02:33:30.921530 31613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:33:30.923359 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 31613
I20260812 02:33:30.923525 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:33:30.924069 12323 external_mini_cluster.cc:1428] Setting key 4dac4bed83efde915627f169b8178572
Aug 12 02:33:30 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502010, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:30.944643 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:30.924151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:48073 (local address 127.12.8.254:43849)
0812 02:33:30.924805 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:30.924812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:30.926036 (+ 1224us) server_negotiation.cc:407] Connection header received
0812 02:33:30.928585 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:30.928590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:30.928662 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:30.928759 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:30.930829 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:30.931621 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:30.933707 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:30.934245 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:30.934357 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:30.938586 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:30.938593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:30.938597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:30.938636 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:30.941409 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:30.942238 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:30.942242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:30.942244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:30.942388 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:30.942886 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:30.942890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:30.942891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:30.943156 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:30.943266 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:30.944102 (+ 836us) server_negotiation.cc:299] Negotiation successful
0812 02:33:30.944275 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":441,"threads_started":1}
I20260812 02:33:30.946094 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:30.924656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:48073)
0812 02:33:30.925859 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:30.925909 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:30.928222 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:30.929037 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:30.929050 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:30.929825 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:30.930625 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:30.930651 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:30.931786 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:30.931791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:30.933556 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:30.933578 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:30.934484 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:30.934492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:30.934773 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:30.935517 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:30.935544 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:30.938312 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:30.941547 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:30.941556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:30.941573 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:30.942090 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:30.942500 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:30.942505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:30.942508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:30.942748 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:30.943277 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:30.943293 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:30.943706 (+ 413us) client_negotiation.cc:769] Sending connection context
0812 02:33:30.944052 (+ 346us) client_negotiation.cc:240] Negotiation successful
0812 02:33:30.944617 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":440,"threads_started":1}
I20260812 02:33:30.947615 31765 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43849
I20260812 02:33:30.947957 31765 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:30.948786 31765 heartbeater.cc:511] Master 127.12.8.254:43849 requested a full tablet report, sending...
I20260812 02:33:30.950587 31374 ts_manager.cc:194] Registered new tserver with Master: 03b245e6a7734141a86af164d77d5dc2 (127.12.8.194:44233)
I20260812 02:33:30.951635 31374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:48073
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:33:30.997357 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43849
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:31.343757 31772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:31.344033 31772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:31.344127 31772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:31.351413 31772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:31.351553 31772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:31.351717 31772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:33:31.361505 31772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:31.362704 31772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:31.363927 31772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:31.377229 31779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:31.377283 31778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:31.378255 31781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:31.378682 31772 server_base.cc:1034] running on GCE node
I20260812 02:33:31.379390 31772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:31.380513 31772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:31.381772 31772 hybrid_clock.cc:648] HybridClock initialized: now 1786502011381722 us; error 29 us; skew 500 ppm
I20260812 02:33:31.382083 31772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:31 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502011, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:31.385299 31772 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:33:31.386852 31772 webserver.cc:460] Webserver started at http://127.12.8.195:36159/ using document root <none> and password file <none>
I20260812 02:33:31.387528 31772 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:31.387650 31772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:31.387970 31772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:33:31.391244 31772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "4b5e28a1cc5f404d87d22e1df63e4c88"
format_stamp: "Formatted at 2026-08-12 02:33:31 on dist-test-slave-vq2q"
server_key: "4d8566434c06ad5580480f4626179360"
server_key_iv: "a819e5e198800b3bda66e713909a8f5b"
server_key_version: "encryptionkey@0"
I20260812 02:33:31.392004 31772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "4b5e28a1cc5f404d87d22e1df63e4c88"
format_stamp: "Formatted at 2026-08-12 02:33:31 on dist-test-slave-vq2q"
server_key: "4d8566434c06ad5580480f4626179360"
server_key_iv: "a819e5e198800b3bda66e713909a8f5b"
server_key_version: "encryptionkey@0"
I20260812 02:33:31.397362 31772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 02:33:31.400849 31788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:31.402127 31772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:33:31.402292 31772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "4b5e28a1cc5f404d87d22e1df63e4c88"
format_stamp: "Formatted at 2026-08-12 02:33:31 on dist-test-slave-vq2q"
server_key: "4d8566434c06ad5580480f4626179360"
server_key_iv: "a819e5e198800b3bda66e713909a8f5b"
server_key_version: "encryptionkey@0"
I20260812 02:33:31.402487 31772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:31.409372 31606 heartbeater.cc:503] Master 127.12.8.254:43849 was elected leader, sending a full tablet report...
I20260812 02:33:31.431679 31772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:31.435866 31772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:31.436131 31772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:31.437101 31772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:31.438685 31772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:31.438764 31772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:31.438860 31772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:31.438905 31772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:31.471208 31772 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:36489
I20260812 02:33:31.471267 31921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:36489 every 8 connection(s)
I20260812 02:33:31.472530 31772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:33:31.478554 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 31772
I20260812 02:33:31.478727 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:33:31.479203 12323 external_mini_cluster.cc:1428] Setting key 67af4c69662c877faa62256c0c3db94a
Aug 12 02:33:31 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502011, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:31.492869 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.475098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:42729 (local address 127.12.8.254:43849)
0812 02:33:31.475568 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.475573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.476358 (+ 785us) server_negotiation.cc:407] Connection header received
0812 02:33:31.478306 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.478313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.478400 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.478501 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:31.480064 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.481045 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.482886 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.483294 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.483395 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.487106 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:31.487112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:31.487115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:31.487158 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:31.489831 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:31.490668 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:31.490672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:31.490675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:31.490812 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:31.491228 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:31.491232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:31.491234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:31.491537 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:31.491665 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.492388 (+ 723us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.492575 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":238,"threads_started":1}
I20260812 02:33:31.494089 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.475499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:42729)
0812 02:33:31.476197 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:31.476238 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:31.478016 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:31.478766 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:31.478774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:31.479313 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:31.479872 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.479895 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.481241 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.481247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.482751 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.482764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.483507 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.483513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.483717 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.484406 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:31.484435 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:31.486850 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:31.489992 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:31.489999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:31.490012 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:31.490502 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:31.490929 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:31.490933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:31.490935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:31.491138 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:31.491662 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:31.491668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:31.492051 (+ 383us) client_negotiation.cc:769] Sending connection context
0812 02:33:31.492293 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 02:33:31.492751 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":272,"threads_started":1}
I20260812 02:33:31.495404 31922 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43849
I20260812 02:33:31.495703 31922 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:31.496404 31922 heartbeater.cc:511] Master 127.12.8.254:43849 requested a full tablet report, sending...
I20260812 02:33:31.497809 31374 ts_manager.cc:194] Registered new tserver with Master: 4b5e28a1cc5f404d87d22e1df63e4c88 (127.12.8.195:36489)
I20260812 02:33:31.498628 31374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:42729
I20260812 02:33:31.507555 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:33:31.526151 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.512660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.12.8.254:43849)
0812 02:33:31.512840 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.512847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.513357 (+ 510us) server_negotiation.cc:407] Connection header received
0812 02:33:31.513556 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.513560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.513630 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.513736 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:31.515403 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.516138 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.517749 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.518204 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.518324 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.520926 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:31.520932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:31.520935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:31.520973 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:31.523411 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:31.524066 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:31.524071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:31.524073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:31.524201 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:31.524568 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:31.524573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:31.524576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:31.524924 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:31.525165 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.525553 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.525835 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:33:31.530509 31374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:33:31.537457 31374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:33:31.561621 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.549587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:36489 (local address 127.0.0.1:36176)
0812 02:33:31.550458 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:31.550507 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:31.550671 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:31.553471 (+ 2800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:31.553480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:31.553510 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:31.553958 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.553972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.556078 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.556084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.558726 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.558741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.560615 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.560621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.560857 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.560892 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 02:33:31.560969 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 02:33:31.561078 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":78,"spinlock_wait_cycles":61312,"thread_start_us":672,"threads_started":1}
I20260812 02:33:31.561681 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.550093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.12.8.194:44233)
0812 02:33:31.552281 (+ 2188us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.552291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.552724 (+ 433us) server_negotiation.cc:407] Connection header received
0812 02:33:31.552899 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.552908 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.553157 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.553380 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:31.554749 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.556200 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.559179 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.560580 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.560709 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.560913 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.561147 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.561360 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2036,"thread_start_us":1931,"threads_started":1}
I20260812 02:33:31.561692 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.552624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.12.8.193:38837)
0812 02:33:31.553310 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.553318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.553961 (+ 643us) server_negotiation.cc:407] Connection header received
0812 02:33:31.554284 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.554291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.554487 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.554658 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:31.555367 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.556604 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.559246 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.560217 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.560351 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.560590 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.561221 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.561398 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":350,"threads_started":1}
I20260812 02:33:31.561643 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.549717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:44233 (local address 127.0.0.1:36628)
0812 02:33:31.552599 (+ 2882us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:31.552618 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:31.552747 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:31.553643 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:31.553654 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:31.554113 (+ 459us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:31.554547 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.554560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.556322 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.556328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.558994 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.559009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.560781 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.560787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.560975 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.561005 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:33:31.561086 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 02:33:31.561234 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2766,"mutex_wait_us":116,"thread_start_us":3447,"threads_started":1}
I20260812 02:33:31.562312 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.551919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:38837 (local address 127.0.0.1:40680)
0812 02:33:31.553849 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:31.553869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:31.554069 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:31.554920 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:31.554925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:31.554948 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:31.555214 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.555222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.556747 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.556752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.559104 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.559133 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.560415 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.560422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.560620 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.560710 (+ 90us) client_negotiation.cc:769] Sending connection context
0812 02:33:31.560839 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 02:33:31.561026 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1812,"mutex_wait_us":94,"thread_start_us":1635,"threads_started":1}
I20260812 02:33:31.562865 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.549867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.12.8.195:36489)
0812 02:33:31.552824 (+ 2957us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.552832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.552869 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:33:31.552964 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.552971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.553231 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.553749 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:31.554141 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.555936 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.558929 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.560372 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.562063 (+ 1691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.562296 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.562378 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.562539 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2743,"thread_start_us":2604,"threads_started":1}
I20260812 02:33:31.565686 31524 tablet_service.cc:1467] Processing CreateTablet for tablet c24bf0f356ce4d919e722109a0215114 (DEFAULT_TABLE table=test-table [id=b0a07690f05d48bfa8a41cc55c02707b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:31.565841 31686 tablet_service.cc:1467] Processing CreateTablet for tablet c24bf0f356ce4d919e722109a0215114 (DEFAULT_TABLE table=test-table [id=b0a07690f05d48bfa8a41cc55c02707b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:31.566182 31845 tablet_service.cc:1467] Processing CreateTablet for tablet c24bf0f356ce4d919e722109a0215114 (DEFAULT_TABLE table=test-table [id=b0a07690f05d48bfa8a41cc55c02707b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:31.567019 31524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24bf0f356ce4d919e722109a0215114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:31.567188 31686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24bf0f356ce4d919e722109a0215114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:31.567428 31845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24bf0f356ce4d919e722109a0215114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:31.578389 31944 tablet_bootstrap.cc:492] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Bootstrap starting.
I20260812 02:33:31.581753 31944 tablet_bootstrap.cc:654] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:31.582773 31944 log.cc:824] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:31.585402 31946 tablet_bootstrap.cc:492] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Bootstrap starting.
I20260812 02:33:31.585609 31944 tablet_bootstrap.cc:492] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: No bootstrap required, opened a new log
I20260812 02:33:31.585912 31945 tablet_bootstrap.cc:492] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: Bootstrap starting.
I20260812 02:33:31.585968 31944 ts_tablet_manager.cc:1397] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 02:33:31.588728 31945 tablet_bootstrap.cc:654] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:31.589331 31946 tablet_bootstrap.cc:654] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:31.590035 31945 log.cc:824] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:31.590622 31946 log.cc:824] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:31.593207 31946 tablet_bootstrap.cc:492] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: No bootstrap required, opened a new log
I20260812 02:33:31.593356 31945 tablet_bootstrap.cc:492] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: No bootstrap required, opened a new log
I20260812 02:33:31.593505 31946 ts_tablet_manager.cc:1397] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260812 02:33:31.593623 31945 ts_tablet_manager.cc:1397] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 02:33:31.596848 31944 raft_consensus.cc:348] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.597360 31944 raft_consensus.cc:374] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:31.597478 31944 raft_consensus.cc:729] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ebadf893584046a0e6b815c049d5d4, State: Initialized, Role: FOLLOWER
I20260812 02:33:31.598151 31944 consensus_queue.cc:260] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.600149 31944 ts_tablet_manager.cc:1428] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260812 02:33:31.602902 31946 raft_consensus.cc:348] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.603462 31946 raft_consensus.cc:374] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:31.603600 31946 raft_consensus.cc:729] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b245e6a7734141a86af164d77d5dc2, State: Initialized, Role: FOLLOWER
I20260812 02:33:31.603955 31945 raft_consensus.cc:348] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.604414 31945 raft_consensus.cc:374] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:31.604275 31946 consensus_queue.cc:260] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.604519 31945 raft_consensus.cc:729] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e28a1cc5f404d87d22e1df63e4c88, State: Initialized, Role: FOLLOWER
I20260812 02:33:31.605165 31945 consensus_queue.cc:260] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.605690 31765 heartbeater.cc:503] Master 127.12.8.254:43849 was elected leader, sending a full tablet report...
I20260812 02:33:31.606333 31922 heartbeater.cc:503] Master 127.12.8.254:43849 was elected leader, sending a full tablet report...
I20260812 02:33:31.606775 31946 ts_tablet_manager.cc:1428] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260812 02:33:31.606973 31945 ts_tablet_manager.cc:1428] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260812 02:33:31.629508 31607 tablet.cc:2369] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:31.629657 31954 raft_consensus.cc:482] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:33:31.630018 31954 raft_consensus.cc:504] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.632215 31954 leader_election.cc:288] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b5e28a1cc5f404d87d22e1df63e4c88 (127.12.8.195:36489), b7ebadf893584046a0e6b815c049d5d4 (127.12.8.193:38837)
I20260812 02:33:31.639101 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.632828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:39747 (local address 127.12.8.195:36489)
0812 02:33:31.633060 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.633066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.633876 (+ 810us) server_negotiation.cc:407] Connection header received
0812 02:33:31.634030 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.634033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.634094 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.634167 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:31.634759 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.635483 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.637452 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.638206 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.638369 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.638458 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.638715 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.638852 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 02:33:31.639196 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.632879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:38329 (local address 127.12.8.193:38837)
0812 02:33:31.633118 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.633125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.633958 (+ 833us) server_negotiation.cc:407] Connection header received
0812 02:33:31.634129 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.634133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.634195 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.634274 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:31.634893 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.635601 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.637455 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.638297 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.638462 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.638538 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.638829 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.638958 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 02:33:31.639257 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.632617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:36489 (local address 127.12.8.194:39747)
0812 02:33:31.633719 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:31.633735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:31.633888 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:31.634191 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:31.634196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:31.634220 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:31.634612 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.634624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.635604 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.635609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.637297 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.637317 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.638419 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.638426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.638563 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.638591 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:33:31.638676 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 02:33:31.638810 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":64,"thread_start_us":787,"threads_started":1}
I20260812 02:33:31.639333 31956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.632791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:38837 (local address 127.12.8.194:38329)
0812 02:33:31.633845 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:31.633860 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:31.633987 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:31.634303 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:31.634307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:31.634348 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:31.634737 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.634747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.635744 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.635748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.637305 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:31.637318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.638507 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.638513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:31.638676 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.638699 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:33:31.638779 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 02:33:31.638898 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":21,"thread_start_us":441,"threads_started":1}
I20260812 02:33:31.640311 31549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24bf0f356ce4d919e722109a0215114" candidate_uuid: "03b245e6a7734141a86af164d77d5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ebadf893584046a0e6b815c049d5d4" is_pre_election: true
I20260812 02:33:31.640331 31867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24bf0f356ce4d919e722109a0215114" candidate_uuid: "03b245e6a7734141a86af164d77d5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" is_pre_election: true
I20260812 02:33:31.640805 31867 raft_consensus.cc:2454] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b245e6a7734141a86af164d77d5dc2 in term 0.
I20260812 02:33:31.640807 31549 raft_consensus.cc:2454] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b245e6a7734141a86af164d77d5dc2 in term 0.
I20260812 02:33:31.641449 31634 leader_election.cc:302] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b245e6a7734141a86af164d77d5dc2, 4b5e28a1cc5f404d87d22e1df63e4c88; no voters:
I20260812 02:33:31.641850 31954 raft_consensus.cc:2781] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:33:31.641979 31954 raft_consensus.cc:482] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:33:31.642077 31954 raft_consensus.cc:3037] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:31.644307 31954 raft_consensus.cc:504] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.644914 31954 leader_election.cc:288] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 4b5e28a1cc5f404d87d22e1df63e4c88 (127.12.8.195:36489), b7ebadf893584046a0e6b815c049d5d4 (127.12.8.193:38837)
I20260812 02:33:31.645390 31867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24bf0f356ce4d919e722109a0215114" candidate_uuid: "03b245e6a7734141a86af164d77d5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88"
I20260812 02:33:31.645452 31549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24bf0f356ce4d919e722109a0215114" candidate_uuid: "03b245e6a7734141a86af164d77d5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ebadf893584046a0e6b815c049d5d4"
I20260812 02:33:31.645574 31867 raft_consensus.cc:3037] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:31.645658 31549 raft_consensus.cc:3037] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:31.647794 31867 raft_consensus.cc:2454] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b245e6a7734141a86af164d77d5dc2 in term 1.
I20260812 02:33:31.647893 31549 raft_consensus.cc:2454] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b245e6a7734141a86af164d77d5dc2 in term 1.
I20260812 02:33:31.648207 31634 leader_election.cc:302] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b245e6a7734141a86af164d77d5dc2, 4b5e28a1cc5f404d87d22e1df63e4c88; no voters:
I20260812 02:33:31.648504 31954 raft_consensus.cc:2781] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:31.648950 31954 raft_consensus.cc:686] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 03b245e6a7734141a86af164d77d5dc2, State: Running, Role: LEADER
I20260812 02:33:31.649487 31954 consensus_queue.cc:237] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:31.654569 31373 catalog_manager.cc:5495] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03b245e6a7734141a86af164d77d5dc2 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "03b245e6a7734141a86af164d77d5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } health_report { overall_health: UNKNOWN } } }
W20260812 02:33:31.673522 31766 tablet.cc:2369] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:31.687458 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.682020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.12.8.194:44233)
0812 02:33:31.682203 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.682208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.682224 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:33:31.682314 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.682318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.682403 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.682522 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:33:31.682992 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.683777 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.685140 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.685678 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.685836 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.686300 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:33:31.686838 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:33:31.686953 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.687136 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.687217 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:33:31.695679 31867 raft_consensus.cc:1261] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 1 FOLLOWER]: Refusing update from remote peer 03b245e6a7734141a86af164d77d5dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:31.695679 31549 raft_consensus.cc:1261] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 1 FOLLOWER]: Refusing update from remote peer 03b245e6a7734141a86af164d77d5dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:31.696798 31959 consensus_queue.cc:1035] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:31.697199 31954 consensus_queue.cc:1035] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:31.712105 31964 mvcc.cc:205] Tried to move back new op lower bound from 7317512239895687168 to 7317512239720984576. Current Snapshot: MvccSnapshot[applied={T|T < 7317512239895687168}]
I20260812 02:33:31.713831 31965 mvcc.cc:205] Tried to move back new op lower bound from 7317512239895687168 to 7317512239720984576. Current Snapshot: MvccSnapshot[applied={T|T < 7317512239895687168}]
I20260812 02:33:31.716166 31966 mvcc.cc:205] Tried to move back new op lower bound from 7317512239895687168 to 7317512239720984576. Current Snapshot: MvccSnapshot[applied={T|T < 7317512239895687168}]
I20260812 02:33:31.738030 31373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:31.738358 31373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:31.742412 31373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Sending DeleteTablet for 3 replicas of tablet c24bf0f356ce4d919e722109a0215114
I20260812 02:33:31.743651 31686 tablet_service.cc:1514] Processing DeleteTablet for tablet c24bf0f356ce4d919e722109a0215114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:31 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:36628
I20260812 02:33:31.743769 31845 tablet_service.cc:1514] Processing DeleteTablet for tablet c24bf0f356ce4d919e722109a0215114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:31 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:36176
I20260812 02:33:31.744014 31524 tablet_service.cc:1514] Processing DeleteTablet for tablet c24bf0f356ce4d919e722109a0215114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:31 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:40680
I20260812 02:33:31.745042 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 31449
I20260812 02:33:31.772836 31977 tablet_replica.cc:330] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: stopping tablet replica
I20260812 02:33:31.772917 31978 tablet_replica.cc:330] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: stopping tablet replica
I20260812 02:33:31.773450 31977 raft_consensus.cc:2229] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:31.773497 31978 raft_consensus.cc:2229] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:31.774084 31977 raft_consensus.cc:2258] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:31.777424 31977 ts_tablet_manager.cc:1905] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:31.783707 31978 raft_consensus.cc:2258] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:31.784098 31976 tablet_replica.cc:330] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: stopping tablet replica
I20260812 02:33:31.784654 31976 raft_consensus.cc:2229] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:33:31.785391 31977 ts_tablet_manager.cc:1918] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:31.785616 31977 log.cc:1196] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/c24bf0f356ce4d919e722109a0215114
I20260812 02:33:31.785635 31976 raft_consensus.cc:2258] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:31.786254 31977 ts_tablet_manager.cc:1939] T c24bf0f356ce4d919e722109a0215114 P 4b5e28a1cc5f404d87d22e1df63e4c88: Deleting consensus metadata
I20260812 02:33:31.788257 31978 ts_tablet_manager.cc:1905] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:31.788264 31359 catalog_manager.cc:4841] TS 4b5e28a1cc5f404d87d22e1df63e4c88 (127.12.8.195:36489): tablet c24bf0f356ce4d919e722109a0215114 (table test-table [id=b0a07690f05d48bfa8a41cc55c02707b]) successfully deleted
I20260812 02:33:31.788384 31976 ts_tablet_manager.cc:1905] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 02:33:31 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:33:31.796525 31976 ts_tablet_manager.cc:1918] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:31.796516 31978 ts_tablet_manager.cc:1918] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:31.796692 31978 log.cc:1196] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/c24bf0f356ce4d919e722109a0215114
I20260812 02:33:31.796887 31976 log.cc:1196] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/c24bf0f356ce4d919e722109a0215114
I20260812 02:33:31.797297 31978 ts_tablet_manager.cc:1939] T c24bf0f356ce4d919e722109a0215114 P b7ebadf893584046a0e6b815c049d5d4: Deleting consensus metadata
I20260812 02:33:31.797537 31976 ts_tablet_manager.cc:1939] T c24bf0f356ce4d919e722109a0215114 P 03b245e6a7734141a86af164d77d5dc2: Deleting consensus metadata
I20260812 02:33:31.799394 31361 catalog_manager.cc:4841] TS b7ebadf893584046a0e6b815c049d5d4 (127.12.8.193:38837): tablet c24bf0f356ce4d919e722109a0215114 (table test-table [id=b0a07690f05d48bfa8a41cc55c02707b]) successfully deleted
I20260812 02:33:31.799357 31357 catalog_manager.cc:4841] TS 03b245e6a7734141a86af164d77d5dc2 (127.12.8.194:44233): tablet c24bf0f356ce4d919e722109a0215114 (table test-table [id=b0a07690f05d48bfa8a41cc55c02707b]) successfully deleted
I20260812 02:33:31.799927 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.746337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.12.8.193:38837)
0812 02:33:31.746525 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.746531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.768780 (+ 22249us) server_negotiation.cc:407] Connection header received
0812 02:33:31.769469 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.769477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.769620 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.769712 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:31.778374 (+ 8662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.779507 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.781611 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.782272 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.782384 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.791497 (+ 9113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:31.791516 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:31.791537 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:31.791594 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:31.795983 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:31.797058 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:31.797069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:31.797073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:31.797275 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:31.797814 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:31.797820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:31.797823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:31.798312 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:31.798625 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.799300 (+ 675us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.799535 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:33:31.801093 31599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:31.893960 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 31449
I20260812 02:33:31.916476 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 31613
Aug 12 02:33:31 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:33:31.936755 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:31.917267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.12.8.194:44233)
0812 02:33:31.917492 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:31.917498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:31.917523 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:33:31.917633 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:31.917639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:31.917733 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:31.917842 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:31.919078 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.920454 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.922616 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:31.923347 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:31.923481 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:31.929282 (+ 5801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:31.929298 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:31.929319 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:31.929377 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:31.933540 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:31.934352 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:31.934362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:31.934366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:31.934554 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:31.934974 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:31.934979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:31.934982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:31.935475 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:31.935817 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:31.936112 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 02:33:31.936363 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:33:31.937603 31759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:32.057135 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 31613
I20260812 02:33:32.078914 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 31772
Aug 12 02:33:32 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:33:32.097599 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:32.079683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.12.8.195:36489)
0812 02:33:32.079876 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:32.079881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:32.079900 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:33:32.080044 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:32.080048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:32.080113 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:32.080205 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:32.081689 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:32.082496 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:32.083760 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:32.084166 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:32.084263 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:32.092171 (+ 7908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:32.092189 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:32.092205 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:32.092244 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:32.094927 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:32.095588 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:32.095596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:32.095599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:32.095751 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:32.096085 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:32.096088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:32.096090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:32.096442 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:32.096687 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:32.096951 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 02:33:32.097140 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:33:32.098256 31915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:32.186432 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 31772
I20260812 02:33:32.207226 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 31337
I20260812 02:33:32.208460 31413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:32.323603 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 31337
I20260812 02:33:52.349756 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43849
--webserver_interface=127.12.8.254
--webserver_port=41331
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:43849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:52.797020 31984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:52.797477 31984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:52.797588 31984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:52.805868 31984 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:33:52.806048 31984 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:33:52.806098 31984 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:33:52.806154 31984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:33:52.806233 31984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:52.806324 31984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:52.806380 31984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:33:52.806447 31984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:33:52.818490 31984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:43849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:52.820382 31984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:52.822208 31984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:52.855921 31991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:52.855940 31994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:52.855966 31990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:52.856382 31984 server_base.cc:1034] running on GCE node
I20260812 02:33:52.858023 31984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:52.860522 31984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:52.861835 31984 hybrid_clock.cc:648] HybridClock initialized: now 1786502032861742 us; error 74 us; skew 500 ppm
I20260812 02:33:52.862375 31984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:52 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502032, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:52.870472 31984 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:33:52.872948 31984 webserver.cc:460] Webserver started at http://127.12.8.254:41331/ using document root <none> and password file <none>
I20260812 02:33:52.874135 31984 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:52.874305 31984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:52.882452 31984 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 02:33:52.886632 32001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:52.888629 31984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:33:52.888906 31984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "141e89072b794cfb95c510528e572f6e"
format_stamp: "Formatted at 2026-08-12 02:33:29 on dist-test-slave-vq2q"
server_key: "0209c0bc22897e0df9f31200d8a8d174"
server_key_iv: "bfe22878fbebd96fe6118354d95b9f60"
server_key_version: "encryptionkey@0"
I20260812 02:33:52.890465 31984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:52.921058 31984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:52.927632 31984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:52.928074 31984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:52.951167 31984 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:43849
I20260812 02:33:52.951200 32067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:43849 every 8 connection(s)
I20260812 02:33:52.953222 31984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:33:52.957966 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 31984
I20260812 02:33:52.958971 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:38837
--local_ip_for_outbound_sockets=127.12.8.193
--tserver_master_addrs=127.12.8.254:43849
--webserver_port=37613
--webserver_interface=127.12.8.193
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:33:52.962581 32068 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:33:52.966771 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Bootstrap starting.
I20260812 02:33:52.992089 32068 log.cc:824] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:53.009701 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:33:53.010468 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Bootstrap complete.
I20260812 02:33:53.021792 32068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:33:53.022925 32068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141e89072b794cfb95c510528e572f6e, State: Initialized, Role: FOLLOWER
I20260812 02:33:53.023769 32068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:33:53.024107 32068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:33:53.024283 32068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:33:53.024500 32068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:33:53.029636 32068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:33:53.030373 32068 leader_election.cc:302] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 141e89072b794cfb95c510528e572f6e; no voters:
I20260812 02:33:53.031168 32068 leader_election.cc:288] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:33:53.031396 32073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:33:53.032807 32073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 2 LEADER]: Becoming Leader. State: Replica: 141e89072b794cfb95c510528e572f6e, State: Running, Role: LEADER
I20260812 02:33:53.033545 32073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:33:53.034169 32068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:33:53.036713 32075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 141e89072b794cfb95c510528e572f6e. Latest consensus state: current_term: 2 leader_uuid: "141e89072b794cfb95c510528e572f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } } }
I20260812 02:33:53.036774 32074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "141e89072b794cfb95c510528e572f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } } }
I20260812 02:33:53.037225 32075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:53.037277 32074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:53.047924 32080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:33:53.062463 32080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:33:53.064344 32080 catalog_manager.cc:1259] Loaded cluster ID: ebe5bee99f51431183725a24be9a3f4e
I20260812 02:33:53.064512 32080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:33:53.069305 32080 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:33:53.071461 32080 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Loaded TSK: 0
I20260812 02:33:53.113508 32080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Generated new TSK 1
I20260812 02:33:53.114159 32080 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:33:53.396653 32070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:53.397011 32070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:53.397152 32070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:53.404495 32070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:53.404619 32070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:53.404780 32070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:33:53.414860 32070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:38837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=37613
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:53.416215 32070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:53.417645 32070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:53.432857 32106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:53.432751 32102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:53.433310 32101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:53.433733 32070 server_base.cc:1034] running on GCE node
I20260812 02:33:53.434365 32070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:53.435417 32070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:53.436637 32070 hybrid_clock.cc:648] HybridClock initialized: now 1786502033436577 us; error 50 us; skew 500 ppm
I20260812 02:33:53.436975 32070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:53 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:53.440114 32070 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:33:53.441661 32070 webserver.cc:460] Webserver started at http://127.12.8.193:37613/ using document root <none> and password file <none>
I20260812 02:33:53.442380 32070 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:53.442519 32070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:53.448025 32070 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:33:53.450942 32116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:53.452234 32070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:33:53.452382 32070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "b7ebadf893584046a0e6b815c049d5d4"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "41a6cddc0b539c48fe4635f9e55b9e78"
server_key_iv: "3c7083d9ce4d99339e712e83abdb6f23"
server_key_version: "encryptionkey@0"
I20260812 02:33:53.453420 32070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:53.468636 32070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:53.472645 32070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:53.472908 32070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:53.473953 32070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:53.475297 32070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:53.475394 32070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:53.475498 32070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:53.475569 32070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:53.508605 32070 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:38837
I20260812 02:33:53.508680 32249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:38837 every 8 connection(s)
I20260812 02:33:53.510255 32070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:33:53.511519 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 32070
I20260812 02:33:53.512193 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:44233
--local_ip_for_outbound_sockets=127.12.8.194
--tserver_master_addrs=127.12.8.254:43849
--webserver_port=38889
--webserver_interface=127.12.8.194
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:33:53 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:53.534466 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:53.513364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:57217 (local address 127.12.8.254:43849)
0812 02:33:53.513960 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:53.513977 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:53.514621 (+ 644us) server_negotiation.cc:407] Connection header received
0812 02:33:53.516790 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:53.516802 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:53.517126 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:53.517483 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:53.519073 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:53.520369 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:53.521969 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:53.522587 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:53.522764 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:53.526275 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:53.526283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:53.526297 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:53.526343 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:53.529382 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:53.530308 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:53.530315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:53.530320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:53.530495 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:53.531128 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:53.531141 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:53.531144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:53.531708 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:53.531864 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:53.532966 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0812 02:33:53.533389 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
I20260812 02:33:53.534894 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:53.513638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:57217)
0812 02:33:53.514434 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:53.514473 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:53.516245 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:53.517646 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:53.517657 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:53.518291 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:53.518839 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:53.518857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:53.520541 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:53.520545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:53.521825 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:53.521839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:53.522826 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:53.522834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:53.523090 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:53.523723 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:53.523748 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:53.526014 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:53.529543 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:53.529552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:53.529567 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:53.530153 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:53.530614 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:53.530618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:53.530621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:53.530860 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:53.531854 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:53.531863 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:53.532256 (+ 393us) client_negotiation.cc:769] Sending connection context
0812 02:33:53.532697 (+ 441us) client_negotiation.cc:240] Negotiation successful
0812 02:33:53.533362 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":341,"threads_started":1}
I20260812 02:33:53.537828 32250 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43849
I20260812 02:33:53.538165 32250 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:53.539122 32250 heartbeater.cc:511] Master 127.12.8.254:43849 requested a full tablet report, sending...
I20260812 02:33:53.541666 32020 ts_manager.cc:194] Registered new tserver with Master: b7ebadf893584046a0e6b815c049d5d4 (127.12.8.193:38837)
I20260812 02:33:53.543143 32020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:57217
W20260812 02:33:53.878595 32256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:53.878875 32256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:53.878970 32256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:53.886302 32256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:53.886412 32256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:53.886535 32256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:33:53.897061 32256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:44233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=38889
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:53.898397 32256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:53.899716 32256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:53.912467 32263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:53.912503 32264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:53.912566 32266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:53.913455 32256 server_base.cc:1034] running on GCE node
I20260812 02:33:53.914007 32256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:53.914965 32256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:53.916196 32256 hybrid_clock.cc:648] HybridClock initialized: now 1786502033916156 us; error 30 us; skew 500 ppm
I20260812 02:33:53.916531 32256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:53 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:53.919819 32256 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:33:53.921437 32256 webserver.cc:460] Webserver started at http://127.12.8.194:38889/ using document root <none> and password file <none>
I20260812 02:33:53.922104 32256 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:53.922225 32256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:53.927742 32256 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 02:33:53.930634 32273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:53.932097 32256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:33:53.932242 32256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "03b245e6a7734141a86af164d77d5dc2"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "678661c7a9c5f4bb7c0ddb43923daf58"
server_key_iv: "5774faa88524a3570b25ac92650e0a63"
server_key_version: "encryptionkey@0"
I20260812 02:33:53.933339 32256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:53.946033 32256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:53.950095 32256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:53.950403 32256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:53.951510 32256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:53.952852 32256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:53.952939 32256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:53.953038 32256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:53.953105 32256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:53.989460 32256 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:44233
I20260812 02:33:53.989531 32408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:44233 every 8 connection(s)
I20260812 02:33:53.990898 32256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:33:53.999547 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 32256
I20260812 02:33:54.000216 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:36489
--local_ip_for_outbound_sockets=127.12.8.195
--tserver_master_addrs=127.12.8.254:43849
--webserver_port=36159
--webserver_interface=127.12.8.195
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:33:53 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:54.011229 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:53.994008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:39141 (local address 127.12.8.254:43849)
0812 02:33:53.994196 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:53.994201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:53.995256 (+ 1055us) server_negotiation.cc:407] Connection header received
0812 02:33:53.997287 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:53.997294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:53.997424 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:53.997522 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:53.999006 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:53.999757 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.001360 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.001802 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.001923 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.005363 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:54.005369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:54.005373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:54.005409 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:54.007933 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.008677 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.008683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.008686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.008837 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.009358 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.009363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.009366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.009694 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:54.009803 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.010700 (+ 897us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.010923 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:33:54.013466 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:53.994417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:39141)
0812 02:33:53.995105 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:53.995151 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:53.996930 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:53.997763 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:53.997771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:53.998275 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:53.998827 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:53.998845 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:53.999904 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:53.999910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.001213 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.001230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.002039 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.002047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.002264 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.002911 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:54.002933 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:54.005099 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:54.008088 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:54.008098 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:54.008116 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:54.008555 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:54.008957 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:54.008961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:54.008964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:54.009230 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:54.009905 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:54.009912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:54.010290 (+ 378us) client_negotiation.cc:769] Sending connection context
0812 02:33:54.011522 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0812 02:33:54.012152 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":258,"threads_started":1}
I20260812 02:33:54.014967 32409 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43849
I20260812 02:33:54.015305 32409 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:54.016135 32409 heartbeater.cc:511] Master 127.12.8.254:43849 requested a full tablet report, sending...
I20260812 02:33:54.017709 32020 ts_manager.cc:194] Registered new tserver with Master: 03b245e6a7734141a86af164d77d5dc2 (127.12.8.194:44233)
I20260812 02:33:54.018548 32020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:39141
W20260812 02:33:54.352336 32412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:54.352664 32412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:54.352805 32412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:54.360276 32412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:54.360422 32412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:54.360580 32412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:33:54.370808 32412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:36489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=36159
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:33:54.372200 32412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:54.373661 32412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:54.387616 32418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:54.387733 32420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:54.387753 32423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:54.389230 32412 server_base.cc:1034] running on GCE node
I20260812 02:33:54.389930 32412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:54.390998 32412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:54.392249 32412 hybrid_clock.cc:648] HybridClock initialized: now 1786502034392171 us; error 75 us; skew 500 ppm
I20260812 02:33:54.392606 32412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:33:54 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:54.396253 32412 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:33:54.398095 32412 webserver.cc:460] Webserver started at http://127.12.8.195:36159/ using document root <none> and password file <none>
I20260812 02:33:54.398823 32412 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:54.398970 32412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:54.405057 32412 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 02:33:54.408357 32431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:54.410100 32412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:33:54.410293 32412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "4b5e28a1cc5f404d87d22e1df63e4c88"
format_stamp: "Formatted at 2026-08-12 02:33:31 on dist-test-slave-vq2q"
server_key: "4d8566434c06ad5580480f4626179360"
server_key_iv: "a819e5e198800b3bda66e713909a8f5b"
server_key_version: "encryptionkey@0"
I20260812 02:33:54.411351 32412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:54.425549 32412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:54.429818 32412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:54.430140 32412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:54.431262 32412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:54.432713 32412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:54.432812 32412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:54.432924 32412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:54.432998 32412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:54.469815 32412 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:36489
I20260812 02:33:54.469899 32563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:36489 every 8 connection(s)
I20260812 02:33:54.471326 32412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:33:54.476699 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 32412
Aug 12 02:33:54 dist-test-slave-vq2q krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:33:54.493773 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.477910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.12.8.254:43849)
0812 02:33:54.478434 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.478439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.479025 (+ 586us) server_negotiation.cc:407] Connection header received
0812 02:33:54.479263 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.479267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.479341 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.479444 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:54.481259 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.482086 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.483417 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.483883 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.483995 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.487397 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:54.487405 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:54.487411 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:54.487450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:54.490346 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.491360 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.491371 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.491374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.491552 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.492050 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.492054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.492056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.492410 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:54.492697 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.493078 (+ 381us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.493353 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":402,"thread_start_us":271,"threads_started":1}
I20260812 02:33:54.497026 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.474708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:39869 (local address 127.12.8.254:43849)
0812 02:33:54.474964 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.474969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.476208 (+ 1239us) server_negotiation.cc:407] Connection header received
0812 02:33:54.479277 (+ 3069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.479284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.479399 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.479500 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:54.481707 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.482584 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.484359 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.484934 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.485096 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.489101 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:54.489106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:54.489110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:54.489150 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:54.492809 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.493879 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.493885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.493888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.494048 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.494616 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.494620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.494622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.494948 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:54.495119 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.496337 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.496638 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":92}
I20260812 02:33:54.498427 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.475033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:39869)
0812 02:33:54.475945 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:54.475997 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:54.478815 (+ 2818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:54.479868 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:54.479879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:54.480600 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:54.481468 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.481494 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.482782 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.482786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.484195 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.484211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.485307 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.485317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.485593 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.486229 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:54.486273 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:54.488834 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:54.492994 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:54.493004 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:54.493023 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:54.493690 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:54.494205 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:54.494210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:54.494213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:54.494489 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:54.495167 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:54.495176 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:54.495809 (+ 633us) client_negotiation.cc:769] Sending connection context
0812 02:33:54.496180 (+ 371us) client_negotiation.cc:240] Negotiation successful
0812 02:33:54.496797 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":344,"threads_started":1}
I20260812 02:33:54.500151 32564 heartbeater.cc:347] Connected to a master server at 127.12.8.254:43849
I20260812 02:33:54.500546 32564 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:54.501544 32564 heartbeater.cc:511] Master 127.12.8.254:43849 requested a full tablet report, sending...
I20260812 02:33:54.503403 32020 ts_manager.cc:194] Registered new tserver with Master: 4b5e28a1cc5f404d87d22e1df63e4c88 (127.12.8.195:36489)
I20260812 02:33:54.504284 32020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:39869
I20260812 02:33:54.510577 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:33:54.518565 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.511780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.12.8.254:43849)
0812 02:33:54.511998 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.512005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.512589 (+ 584us) server_negotiation.cc:407] Connection header received
0812 02:33:54.512809 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.512814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.513000 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.513220 (+ 220us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:33:54.513660 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.514788 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.516699 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.517139 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.517283 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.517815 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:33:54.518074 (+ 259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 02:33:54.518375 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41518: token expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 02:33:54.518682 32253 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41518: token expired
W20260812 02:33:54.519729 31927 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.8.254:43849)
I20260812 02:33:54.533476 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.520340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.12.8.254:43849)
0812 02:33:54.520542 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.520548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.520639 (+ 91us) server_negotiation.cc:407] Connection header received
0812 02:33:54.520835 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.520841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.520925 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.521005 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:54.522515 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.523266 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.524592 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.525008 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.525116 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.527889 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:54.527895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:54.527898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:54.527932 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:54.530700 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.531446 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.531451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.531455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.531578 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.531944 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.531948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.531950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.532285 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:54.532525 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.532858 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.533094 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:33:54.538218 32020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:33:54.546906 32020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:33:54.547011 32250 heartbeater.cc:503] Master 127.12.8.254:43849 was elected leader, sending a full tablet report...
I20260812 02:33:54.600087 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.563287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:36489 (local address 127.0.0.1:50776)
0812 02:33:54.588015 (+ 24728us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:54.588050 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:54.588315 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:54.591335 (+ 3020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:54.591340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:54.591361 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:54.591836 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.591848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.594394 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.594401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.597288 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.597308 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.598916 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.598924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.599132 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.599173 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 02:33:54.599313 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 02:33:54.599547 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24475,"mutex_wait_us":26,"thread_start_us":25994,"threads_started":1}
I20260812 02:33:54.600114 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.566401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:38837 (local address 127.0.0.1:35508)
0812 02:33:54.589613 (+ 23212us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:54.589632 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:54.589787 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:54.591008 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:54.591016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:54.591066 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:54.591592 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.591607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.594313 (+ 2706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.594319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.597007 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.597024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.599338 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.599343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.599517 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.599547 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:33:54.599612 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 02:33:54.599736 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23077,"thread_start_us":22049,"threads_started":1}
I20260812 02:33:54.600916 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.563519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.12.8.195:36489)
0812 02:33:54.590531 (+ 27012us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.590541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.590576 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:33:54.590723 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.590732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.591073 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.591359 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:54.592090 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.594185 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.597550 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.598662 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.598792 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.599050 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.600400 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.600589 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26777,"thread_start_us":28139,"threads_started":1}
I20260812 02:33:54.601254 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.562759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:44233 (local address 127.0.0.1:41064)
0812 02:33:54.579545 (+ 16786us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:54.579654 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:54.580006 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:54.592741 (+ 12735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:54.592746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:54.592779 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:54.593121 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.593132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.594857 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.594868 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.597546 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.597561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.600264 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.600271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.600458 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.600493 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 02:33:54.600564 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 02:33:54.600745 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16302,"mutex_wait_us":95,"thread_start_us":16024,"threads_started":1}
I20260812 02:33:54.602119 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.566782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.12.8.193:38837)
0812 02:33:54.586130 (+ 19348us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.586144 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.589811 (+ 3667us) server_negotiation.cc:407] Connection header received
0812 02:33:54.590396 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.590405 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.590732 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.590969 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:54.591886 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.593843 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.597284 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.599077 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.599276 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.599575 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.599704 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.601733 (+ 2029us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19088,"thread_start_us":19375,"threads_started":1}
I20260812 02:33:54.602631 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.562996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.12.8.194:44233)
0812 02:33:54.592265 (+ 29269us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.592272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.592294 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:33:54.592368 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.592373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.592526 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.592696 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:54.593311 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.594680 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.597774 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.599988 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.601792 (+ 1804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.602056 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.602150 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.602371 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29116,"thread_start_us":28997,"threads_started":1}
I20260812 02:33:54.604954 32488 tablet_service.cc:1467] Processing CreateTablet for tablet 54f063c6a1104fafab0561e54a712d67 (DEFAULT_TABLE table=test-table [id=d72bd1a8239947cdb3e99251e74e7466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:54.606698 32488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f063c6a1104fafab0561e54a712d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:54.606216 32173 tablet_service.cc:1467] Processing CreateTablet for tablet 54f063c6a1104fafab0561e54a712d67 (DEFAULT_TABLE table=test-table [id=d72bd1a8239947cdb3e99251e74e7466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:54.606537 32329 tablet_service.cc:1467] Processing CreateTablet for tablet 54f063c6a1104fafab0561e54a712d67 (DEFAULT_TABLE table=test-table [id=d72bd1a8239947cdb3e99251e74e7466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:54.608006 32173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f063c6a1104fafab0561e54a712d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:54.608017 32329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f063c6a1104fafab0561e54a712d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:54.623965 32583 tablet_bootstrap.cc:492] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Bootstrap starting.
I20260812 02:33:54.625229 32585 tablet_bootstrap.cc:492] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Bootstrap starting.
I20260812 02:33:54.629966 32586 tablet_bootstrap.cc:492] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Bootstrap starting.
I20260812 02:33:54.630687 32585 tablet_bootstrap.cc:654] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:54.632676 32585 log.cc:824] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:54.633795 32583 tablet_bootstrap.cc:654] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:54.634457 32586 tablet_bootstrap.cc:654] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:54.635210 32583 log.cc:824] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:54.635900 32586 log.cc:824] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:54.637632 32585 tablet_bootstrap.cc:492] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: No bootstrap required, opened a new log
I20260812 02:33:54.638561 32585 ts_tablet_manager.cc:1397] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260812 02:33:54.642715 32583 tablet_bootstrap.cc:492] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: No bootstrap required, opened a new log
I20260812 02:33:54.643172 32583 ts_tablet_manager.cc:1397] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20260812 02:33:54.646826 32586 tablet_bootstrap.cc:492] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: No bootstrap required, opened a new log
I20260812 02:33:54.647447 32586 ts_tablet_manager.cc:1397] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20260812 02:33:54.651815 32585 raft_consensus.cc:348] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:54.652534 32585 raft_consensus.cc:374] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:54.652660 32585 raft_consensus.cc:729] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b245e6a7734141a86af164d77d5dc2, State: Initialized, Role: FOLLOWER
I20260812 02:33:54.653069 32583 raft_consensus.cc:348] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:54.653612 32583 raft_consensus.cc:374] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:54.653724 32583 raft_consensus.cc:729] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e28a1cc5f404d87d22e1df63e4c88, State: Initialized, Role: FOLLOWER
I20260812 02:33:54.653612 32585 consensus_queue.cc:260] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:54.654413 32583 consensus_queue.cc:260] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:54.658023 32409 heartbeater.cc:503] Master 127.12.8.254:43849 was elected leader, sending a full tablet report...
I20260812 02:33:54.658257 32564 heartbeater.cc:503] Master 127.12.8.254:43849 was elected leader, sending a full tablet report...
I20260812 02:33:54.658790 32585 ts_tablet_manager.cc:1428] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260812 02:33:54.660470 32583 ts_tablet_manager.cc:1428] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260812 02:33:54.662189 32586 raft_consensus.cc:348] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:54.662856 32586 raft_consensus.cc:374] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:54.662990 32586 raft_consensus.cc:729] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ebadf893584046a0e6b815c049d5d4, State: Initialized, Role: FOLLOWER
I20260812 02:33:54.663956 32586 consensus_queue.cc:260] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:54.667309 32586 ts_tablet_manager.cc:1428] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Time spent starting tablet: real 0.020s user 0.014s sys 0.002s
I20260812 02:33:54.713128 32591 raft_consensus.cc:482] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:33:54.713641 32591 raft_consensus.cc:504] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:54.715932 32591 leader_election.cc:288] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b5e28a1cc5f404d87d22e1df63e4c88 (127.12.8.195:36489), b7ebadf893584046a0e6b815c049d5d4 (127.12.8.193:38837)
W20260812 02:33:54.723220 32565 tablet.cc:2369] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:54.725356 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.716583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:37015 (local address 127.12.8.195:36489)
0812 02:33:54.716874 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.716881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.717992 (+ 1111us) server_negotiation.cc:407] Connection header received
0812 02:33:54.718191 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.718197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.718282 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.718376 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:54.719136 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.719993 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.722643 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.723893 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.724112 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.724199 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.724700 (+ 501us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.724901 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260812 02:33:54.725376 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.716704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:44159 (local address 127.12.8.193:38837)
0812 02:33:54.716969 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.716975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.718195 (+ 1220us) server_negotiation.cc:407] Connection header received
0812 02:33:54.718399 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.718405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.718498 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.718612 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:54.719386 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.720290 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.722637 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.723686 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.723898 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.724007 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.724614 (+ 607us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.724817 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 02:33:54.725381 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.716488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:36489 (local address 127.12.8.194:37015)
0812 02:33:54.717833 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:54.717853 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:54.718032 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:54.718410 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:54.718415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:54.718443 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:54.718959 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.718973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.720154 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.720159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.722416 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.722437 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.724244 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.724250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.724463 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.724493 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:33:54.724580 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 02:33:54.724716 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":149,"thread_start_us":761,"threads_started":1}
I20260812 02:33:54.725402 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.716704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:38837 (local address 127.12.8.194:44159)
0812 02:33:54.718035 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:54.718056 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:54.718239 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:54.718647 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:54.718651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:54.718679 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:54.719182 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.719197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.720470 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.720474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.722418 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:54.722434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.723990 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.723996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:54.724321 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.724367 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 02:33:54.724494 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 02:33:54.724710 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":168,"thread_start_us":821,"threads_started":1}
I20260812 02:33:54.726622 32508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f063c6a1104fafab0561e54a712d67" candidate_uuid: "03b245e6a7734141a86af164d77d5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" is_pre_election: true
I20260812 02:33:54.727193 32508 raft_consensus.cc:2454] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b245e6a7734141a86af164d77d5dc2 in term 0.
I20260812 02:33:54.727413 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f063c6a1104fafab0561e54a712d67" candidate_uuid: "03b245e6a7734141a86af164d77d5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ebadf893584046a0e6b815c049d5d4" is_pre_election: true
I20260812 02:33:54.728041 32275 leader_election.cc:302] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b245e6a7734141a86af164d77d5dc2, 4b5e28a1cc5f404d87d22e1df63e4c88; no voters:
I20260812 02:33:54.728166 32197 raft_consensus.cc:2454] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b245e6a7734141a86af164d77d5dc2 in term 0.
I20260812 02:33:54.728540 32591 raft_consensus.cc:2781] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:33:54.728682 32591 raft_consensus.cc:482] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:33:54.728801 32591 raft_consensus.cc:3037] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:54.731313 32591 raft_consensus.cc:504] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
I20260812 02:33:54.732029 32591 leader_election.cc:288] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 4b5e28a1cc5f404d87d22e1df63e4c88 (127.12.8.195:36489), b7ebadf893584046a0e6b815c049d5d4 (127.12.8.193:38837)
I20260812 02:33:54.732890 32508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f063c6a1104fafab0561e54a712d67" candidate_uuid: "03b245e6a7734141a86af164d77d5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88"
I20260812 02:33:54.732980 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f063c6a1104fafab0561e54a712d67" candidate_uuid: "03b245e6a7734141a86af164d77d5dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ebadf893584046a0e6b815c049d5d4"
I20260812 02:33:54.733114 32508 raft_consensus.cc:3037] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:54.733311 32197 raft_consensus.cc:3037] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:54.735643 32508 raft_consensus.cc:2454] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b245e6a7734141a86af164d77d5dc2 in term 1.
I20260812 02:33:54.736245 32275 leader_election.cc:302] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b245e6a7734141a86af164d77d5dc2, 4b5e28a1cc5f404d87d22e1df63e4c88; no voters:
I20260812 02:33:54.736255 32197 raft_consensus.cc:2454] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b245e6a7734141a86af164d77d5dc2 in term 1.
I20260812 02:33:54.736578 32591 raft_consensus.cc:2781] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:54.737128 32591 raft_consensus.cc:686] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 03b245e6a7734141a86af164d77d5dc2, State: Running, Role: LEADER
I20260812 02:33:54.737757 32591 consensus_queue.cc:237] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } }
W20260812 02:33:54.742947 32410 tablet.cc:2369] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:54.743327 32020 catalog_manager.cc:5495] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03b245e6a7734141a86af164d77d5dc2 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "03b245e6a7734141a86af164d77d5dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b245e6a7734141a86af164d77d5dc2" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 } health_report { overall_health: UNKNOWN } } }
W20260812 02:33:54.762884 32251 tablet.cc:2369] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:33:54.795965 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.789681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.12.8.194:44233)
0812 02:33:54.789906 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.789911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.789933 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:33:54.790095 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.790100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.790204 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.790395 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:33:54.790815 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.791707 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.793302 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.793834 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.793966 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.794572 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:33:54.795244 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:33:54.795379 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.795553 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.795659 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:33:54.806092 32508 raft_consensus.cc:1261] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 1 FOLLOWER]: Refusing update from remote peer 03b245e6a7734141a86af164d77d5dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:54.806257 32197 raft_consensus.cc:1261] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 1 FOLLOWER]: Refusing update from remote peer 03b245e6a7734141a86af164d77d5dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:54.807173 32596 consensus_queue.cc:1035] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e28a1cc5f404d87d22e1df63e4c88" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:54.807614 32591 consensus_queue.cc:1035] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ebadf893584046a0e6b815c049d5d4" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 38837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:54.843645 32019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:54.844075 32019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:54.848917 32019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Sending DeleteTablet for 3 replicas of tablet 54f063c6a1104fafab0561e54a712d67
I20260812 02:33:54.850713 32488 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f063c6a1104fafab0561e54a712d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:54 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:50776
I20260812 02:33:54.850697 32329 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f063c6a1104fafab0561e54a712d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:54 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:41064
I20260812 02:33:54.851027 32173 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f063c6a1104fafab0561e54a712d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:54 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:35508
I20260812 02:33:54.852386 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 32070
I20260812 02:33:54.862461 32611 tablet_replica.cc:330] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: stopping tablet replica
I20260812 02:33:54.863521 32611 raft_consensus.cc:2229] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:33:54.864632 32612 tablet_replica.cc:330] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: stopping tablet replica
I20260812 02:33:54.865294 32612 raft_consensus.cc:2229] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:54.865360 32611 raft_consensus.cc:2258] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:54.866080 32612 raft_consensus.cc:2258] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:54.866458 32610 tablet_replica.cc:330] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: stopping tablet replica
I20260812 02:33:54.866968 32610 raft_consensus.cc:2229] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:54.867513 32610 raft_consensus.cc:2258] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:54.868602 32611 ts_tablet_manager.cc:1905] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:54.870869 32612 ts_tablet_manager.cc:1905] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:54.872514 32610 ts_tablet_manager.cc:1905] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:54.876787 32611 ts_tablet_manager.cc:1918] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:54.876998 32611 log.cc:1196] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/54f063c6a1104fafab0561e54a712d67
I20260812 02:33:54.877627 32611 ts_tablet_manager.cc:1939] T 54f063c6a1104fafab0561e54a712d67 P 03b245e6a7734141a86af164d77d5dc2: Deleting consensus metadata
I20260812 02:33:54.879596 32612 ts_tablet_manager.cc:1918] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:54.879923 32612 log.cc:1196] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/54f063c6a1104fafab0561e54a712d67
I20260812 02:33:54.879961 32610 ts_tablet_manager.cc:1918] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:54.880180 32610 log.cc:1196] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/54f063c6a1104fafab0561e54a712d67
I20260812 02:33:54.880323 32003 catalog_manager.cc:4841] TS 03b245e6a7734141a86af164d77d5dc2 (127.12.8.194:44233): tablet 54f063c6a1104fafab0561e54a712d67 (table test-table [id=d72bd1a8239947cdb3e99251e74e7466]) successfully deleted
I20260812 02:33:54.880721 32612 ts_tablet_manager.cc:1939] T 54f063c6a1104fafab0561e54a712d67 P b7ebadf893584046a0e6b815c049d5d4: Deleting consensus metadata
I20260812 02:33:54.880823 32610 ts_tablet_manager.cc:1939] T 54f063c6a1104fafab0561e54a712d67 P 4b5e28a1cc5f404d87d22e1df63e4c88: Deleting consensus metadata
I20260812 02:33:54.883077 32004 catalog_manager.cc:4841] TS 4b5e28a1cc5f404d87d22e1df63e4c88 (127.12.8.195:36489): tablet 54f063c6a1104fafab0561e54a712d67 (table test-table [id=d72bd1a8239947cdb3e99251e74e7466]) successfully deleted
I20260812 02:33:54.883422 32006 catalog_manager.cc:4841] TS b7ebadf893584046a0e6b815c049d5d4 (127.12.8.193:38837): tablet 54f063c6a1104fafab0561e54a712d67 (table test-table [id=d72bd1a8239947cdb3e99251e74e7466]) successfully deleted
I20260812 02:33:54.885398 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:54.854309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.12.8.193:38837)
0812 02:33:54.854587 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:54.854596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:54.854620 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:33:54.854702 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:54.854709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:54.854959 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:54.855108 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:54.857514 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.858889 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.861125 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:54.861947 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:54.862072 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:54.867107 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:54.867128 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:54.867146 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:54.867193 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:54.881423 (+ 14230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.882830 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.882841 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.882844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.883134 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:54.883630 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:54.883635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:54.883638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:54.884080 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:54.884393 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:54.884717 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 02:33:54.884973 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 02:33:54.886338 32242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:54.994681 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 32070
I20260812 02:33:55.024281 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 32256
I20260812 02:33:55.043076 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:55.025323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.12.8.194:44233)
0812 02:33:55.025610 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:55.025618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:55.025651 (+ 33us) server_negotiation.cc:407] Connection header received
0812 02:33:55.025766 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:55.025772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:55.025885 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:55.026020 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:55.027800 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:55.029102 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:55.030782 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:55.031510 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:55.031669 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:55.034797 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:55.034820 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:55.034847 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:55.034906 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:55.039515 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:55.040411 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:55.040421 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:55.040425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:55.040647 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:55.041138 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:55.041143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:55.041147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:55.041759 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:55.042147 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:55.042341 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 02:33:55.042622 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 02:33:55.044159 32403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:55.165144 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 32256
I20260812 02:33:55.195161 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 32412
I20260812 02:33:55.211615 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:55.195978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.12.8.195:36489)
0812 02:33:55.196258 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:55.196265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:55.196318 (+ 53us) server_negotiation.cc:407] Connection header received
0812 02:33:55.196560 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:55.196567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:55.196660 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:55.196787 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:55.198300 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:55.199333 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:55.200914 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:55.201924 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:55.202108 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:55.204909 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:55.204931 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:55.204953 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:55.205004 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:55.208290 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:55.209295 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:55.209307 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:55.209311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:55.209499 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:55.209855 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:55.209860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:55.209863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:55.210359 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:55.210623 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:55.210992 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 02:33:55.211246 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 02:33:55.212404 32557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:55.304260 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 32412
I20260812 02:33:55.323809 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 31984
I20260812 02:33:55.325002 32060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:33:55.444033 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 31984
I20260812 02:34:15.470172 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43849
--webserver_interface=127.12.8.254
--webserver_port=41331
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:43849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:15.894515 32619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:15.894847 32619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:15.894944 32619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:15.902498 32619 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:34:15.902624 32619 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:34:15.902663 32619 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:34:15.902709 32619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:15.902768 32619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:15.902845 32619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:15.902899 32619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:15.902957 32619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:15.913585 32619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:43849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:15.915010 32619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:15.916460 32619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:15.930394 32629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:15.930759 32632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:15.930965 32630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:15.937523 32619 server_base.cc:1034] running on GCE node
I20260812 02:34:15.938577 32619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:15.940321 32619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:15.941581 32619 hybrid_clock.cc:648] HybridClock initialized: now 1786502055941502 us; error 58 us; skew 500 ppm
I20260812 02:34:15.941973 32619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:15 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502055, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:15.950136 32619 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:34:15.952057 32619 webserver.cc:460] Webserver started at http://127.12.8.254:41331/ using document root <none> and password file <none>
I20260812 02:34:15.952934 32619 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:15.953053 32619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:15.959728 32619 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:34:15.963371 32639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:15.965148 32619 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:34:15.965420 32619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "141e89072b794cfb95c510528e572f6e"
format_stamp: "Formatted at 2026-08-12 02:33:29 on dist-test-slave-vq2q"
server_key: "0209c0bc22897e0df9f31200d8a8d174"
server_key_iv: "bfe22878fbebd96fe6118354d95b9f60"
server_key_version: "encryptionkey@0"
I20260812 02:34:15.966537 32619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:15.979516 32619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:15.984082 32619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:15.984390 32619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:16.005302 32619 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:43849
I20260812 02:34:16.005391 32698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:43849 every 8 connection(s)
I20260812 02:34:16.006919 32619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:16.009899 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 32619
I20260812 02:34:16.011171 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:38837
--local_ip_for_outbound_sockets=127.12.8.193
--tserver_master_addrs=127.12.8.254:43849
--webserver_port=37613
--webserver_interface=127.12.8.193
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:16.018637 32699 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:34:16.024650 32699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Bootstrap starting.
I20260812 02:34:16.054288 32699 log.cc:824] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:16.075994 32699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:34:16.076802 32699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Bootstrap complete.
I20260812 02:34:16.088678 32699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:34:16.089954 32699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141e89072b794cfb95c510528e572f6e, State: Initialized, Role: FOLLOWER
I20260812 02:34:16.090847 32699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:34:16.091147 32699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:16.091280 32699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:16.091455 32699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:34:16.096518 32699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:34:16.097260 32699 leader_election.cc:302] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 141e89072b794cfb95c510528e572f6e; no voters:
I20260812 02:34:16.098063 32699 leader_election.cc:288] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 02:34:16.098249 32704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:34:16.099552 32704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [term 3 LEADER]: Becoming Leader. State: Replica: 141e89072b794cfb95c510528e572f6e, State: Running, Role: LEADER
I20260812 02:34:16.100368 32704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } }
I20260812 02:34:16.100893 32699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:16.105022 32708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 141e89072b794cfb95c510528e572f6e. Latest consensus state: current_term: 3 leader_uuid: "141e89072b794cfb95c510528e572f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } } }
I20260812 02:34:16.105569 32708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:16.106279 32707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "141e89072b794cfb95c510528e572f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141e89072b794cfb95c510528e572f6e" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43849 } } }
I20260812 02:34:16.106608 32707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:16.117219 32715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:16.130812 32715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:16.133247 32715 catalog_manager.cc:1259] Loaded cluster ID: ebe5bee99f51431183725a24be9a3f4e
I20260812 02:34:16.133543 32715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:16.139345 32715 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:16.147236 32715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Loaded TSK: 0
I20260812 02:34:16.147675 32715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 141e89072b794cfb95c510528e572f6e: Loaded TSK: 1
F20260812 02:34:16.149474 32715 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f19a24689ad google::LogMessage::Fail() at ??:0
@ 0x7f19a246b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f19a24684de google::LogMessage::Flush() at ??:0
@ 0x7f19a2469a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f19a61cc7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f19a61bb73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f19b7e971c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f19b7e96783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f19b7f222a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f19b7f220c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f19b7f6c5e9 std::function<>::operator()() at ??:0
@ 0x7f19b7e9acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f19b7e98c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f19b7f20610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f19b7f20439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f19a50b4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f19a50b791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f19a50b7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f19a5092eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f19a836c6db start_thread at ??:0
@ 0x7f19a130261f clone at ??:0
W20260812 02:34:16.471940 32701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:16.472270 32701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:16.472384 32701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:16.480207 32701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:16.480355 32701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:16.480515 32701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:34:16.491003 32701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:38837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=37613
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:16.492496 32701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:16.494000 32701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:16.510087 32762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:16.510238 32758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:16.510913 32759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:16.511451 32701 server_base.cc:1034] running on GCE node
I20260812 02:34:16.512142 32701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:16.513316 32701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:16.514566 32701 hybrid_clock.cc:648] HybridClock initialized: now 1786502056514486 us; error 76 us; skew 500 ppm
I20260812 02:34:16.514961 32701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:16 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502056, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:16.518756 32701 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:34:16.520663 32701 webserver.cc:460] Webserver started at http://127.12.8.193:37613/ using document root <none> and password file <none>
I20260812 02:34:16.521535 32701 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:16.521672 32701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:16.527940 32701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:34:16.531682 303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:16.533406 32701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:34:16.533600 32701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "b7ebadf893584046a0e6b815c049d5d4"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "41a6cddc0b539c48fe4635f9e55b9e78"
server_key_iv: "3c7083d9ce4d99339e712e83abdb6f23"
server_key_version: "encryptionkey@0"
I20260812 02:34:16.534721 32701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:16.563544 32701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:16.568058 32701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:16.568318 32701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:16.569577 32701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:16.571125 32701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:16.571220 32701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:16.571310 32701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:16.571378 32701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:16.609921 32701 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:38837
I20260812 02:34:16.610163 436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:38837 every 8 connection(s)
I20260812 02:34:16.611528 32701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:16.613714 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 32701
I20260812 02:34:16.614389 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:44233
--local_ip_for_outbound_sockets=127.12.8.194
--tserver_master_addrs=127.12.8.254:43849
--webserver_port=38889
--webserver_interface=127.12.8.194
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:16.618429 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:16.615610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:55357)
0812 02:34:16.616742 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:16.617142 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":485,"threads_started":1}
W20260812 02:34:16.619519 437 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:43849 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.254:43849: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
I20260812 02:34:16.620738 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:16.620296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:52813)
0812 02:34:16.620487 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:16.620578 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 02:34:16.621812 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:16.621420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:39439)
0812 02:34:16.621577 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:16.621665 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 02:34:16.622069 437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:34:17.002344 442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:17.002749 442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:17.002897 442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:17.010726 442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:17.010861 442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:17.011022 442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:34:17.021229 442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:44233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=38889
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:17.022630 442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:17.024104 442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:17.039717 453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:17.039813 451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:17.039848 450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:17.040630 442 server_base.cc:1034] running on GCE node
I20260812 02:34:17.041373 442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:17.042418 442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:17.043648 442 hybrid_clock.cc:648] HybridClock initialized: now 1786502057043595 us; error 49 us; skew 500 ppm
I20260812 02:34:17.044015 442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:17 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502057, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:17.047365 442 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:34:17.049031 442 webserver.cc:460] Webserver started at http://127.12.8.194:38889/ using document root <none> and password file <none>
I20260812 02:34:17.049794 442 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:17.049916 442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:17.055956 442 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:34:17.059247 460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:17.060961 442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:34:17.061151 442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "03b245e6a7734141a86af164d77d5dc2"
format_stamp: "Formatted at 2026-08-12 02:33:30 on dist-test-slave-vq2q"
server_key: "678661c7a9c5f4bb7c0ddb43923daf58"
server_key_iv: "5774faa88524a3570b25ac92650e0a63"
server_key_version: "encryptionkey@0"
I20260812 02:34:17.062245 442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:17.078399 442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:17.082736 442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:17.083014 442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:17.084156 442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:17.085774 442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:17.085879 442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:17.085991 442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:17.086077 442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:17.124493 442 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:44233
I20260812 02:34:17.124630 598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:44233 every 8 connection(s)
I20260812 02:34:17.126104 442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:17.132164 601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:17.129831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:50007)
0812 02:34:17.130822 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:17.131098 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":380,"threads_started":1}
W20260812 02:34:17.133230 599 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:43849 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.254:43849: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
I20260812 02:34:17.134352 601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:17.133941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:60331)
0812 02:34:17.134135 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:17.134229 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 02:34:17.135394 601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:17.135034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:37647)
0812 02:34:17.135184 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:17.135271 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 02:34:17.135651 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 442
W20260812 02:34:17.135726 599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:34:17.136466 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:36489
--local_ip_for_outbound_sockets=127.12.8.195
--tserver_master_addrs=127.12.8.254:43849
--webserver_port=36159
--webserver_interface=127.12.8.195
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:17.506412 602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:17.506778 602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:17.506929 602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:17.514423 602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:17.514561 602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:17.514714 602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:34:17.524523 602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:36489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=36159
--tserver_master_addrs=127.12.8.254:43849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:17.525936 602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:17.527248 602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:17.542495 611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:17.544059 612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:17.544152 614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:17.544183 602 server_base.cc:1034] running on GCE node
I20260812 02:34:17.544986 602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:17.546119 602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:17.547348 602 hybrid_clock.cc:648] HybridClock initialized: now 1786502057547273 us; error 61 us; skew 500 ppm
I20260812 02:34:17.547690 602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:17 dist-test-slave-vq2q krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502057, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:17.551074 602 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:34:17.552767 602 webserver.cc:460] Webserver started at http://127.12.8.195:36159/ using document root <none> and password file <none>
I20260812 02:34:17.553565 602 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:17.553686 602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:17.559762 602 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:34:17.563175 624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:17.564955 602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 02:34:17.565160 602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "4b5e28a1cc5f404d87d22e1df63e4c88"
format_stamp: "Formatted at 2026-08-12 02:33:31 on dist-test-slave-vq2q"
server_key: "4d8566434c06ad5580480f4626179360"
server_key_iv: "a819e5e198800b3bda66e713909a8f5b"
server_key_version: "encryptionkey@0"
I20260812 02:34:17.566289 602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:17.578660 602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:17.583063 602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:17.583343 602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:17.584695 602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:17.586328 602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:17.586437 602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:17.586549 602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:17.586625 602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:17.624109 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:17.622893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:57861)
0812 02:34:17.623614 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:17.623777 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":326,"threads_started":1}
I20260812 02:34:17.624751 602 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:36489
I20260812 02:34:17.624846 754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:36489 every 8 connection(s)
I20260812 02:34:17.626318 602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:17.627892 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 602
I20260812 02:34:17.632810 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:17.630254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:51815)
0812 02:34:17.631262 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:17.631566 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":381,"threads_started":1}
W20260812 02:34:17.633808 756 heartbeater.cc:644] Failed to heartbeat to 127.12.8.254:43849 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.254:43849: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
I20260812 02:34:17.634868 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:17.634480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:52471)
0812 02:34:17.634649 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:17.634737 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 02:34:17.635782 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:17.635450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:34289)
0812 02:34:17.635575 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:17.635659 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:34:17.636127 756 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:34:18.137614 760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:18.136543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:44101)
0812 02:34:18.137139 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:18.137317 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":277,"threads_started":1}
I20260812 02:34:18.626382 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:18.625450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:41697)
0812 02:34:18.626066 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:18.626168 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":324,"threads_started":1}
I20260812 02:34:18.638070 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:18.636975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:48271)
0812 02:34:18.637661 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:18.637763 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":334,"threads_started":1}
I20260812 02:34:19.139781 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:19.138826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:53825)
0812 02:34:19.139427 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:19.139535 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":298,"threads_started":1}
I20260812 02:34:19.628289 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:19.627392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:43559)
0812 02:34:19.627955 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:19.628066 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":311,"threads_started":1}
I20260812 02:34:19.640100 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:19.639173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:40477)
0812 02:34:19.639736 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:19.639824 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":292,"threads_started":1}
I20260812 02:34:20.141723 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:20.140822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:44563)
0812 02:34:20.141385 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:20.141479 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":290,"threads_started":1}
I20260812 02:34:20.630537 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:20.629572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:55019)
0812 02:34:20.630171 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:20.630301 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":322,"threads_started":1}
I20260812 02:34:20.642298 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:20.641243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:57891)
0812 02:34:20.641947 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:20.642063 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":387,"threads_started":1}
I20260812 02:34:21.143826 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:21.142930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:52869)
0812 02:34:21.143476 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:21.143579 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":280,"threads_started":1}
I20260812 02:34:21.632460 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:21.631663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:53749)
0812 02:34:21.632149 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:21.632250 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260812 02:34:21.644285 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:21.643299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:50847)
0812 02:34:21.643897 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:21.644009 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":311,"threads_started":1}
I20260812 02:34:22.146025 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:22.145026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:46719)
0812 02:34:22.145676 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:22.145812 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":312,"threads_started":1}
I20260812 02:34:22.634397 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:22.633515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:36521)
0812 02:34:22.634052 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:22.634182 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":268,"threads_started":1}
I20260812 02:34:22.646390 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:22.645433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:43945)
0812 02:34:22.646041 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:22.646135 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":293,"threads_started":1}
I20260812 02:34:23.148283 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:23.147398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:60123)
0812 02:34:23.147976 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:23.148104 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":281,"threads_started":1}
I20260812 02:34:23.636571 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:23.635462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:52213)
0812 02:34:23.636180 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:23.636301 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":290,"threads_started":1}
I20260812 02:34:23.648504 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:23.647656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:34731)
0812 02:34:23.648206 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:23.648309 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":305,"threads_started":1}
I20260812 02:34:24.150831 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:24.149738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:37741)
0812 02:34:24.150400 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:24.150513 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":278,"threads_started":1}
I20260812 02:34:24.639175 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:24.638098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:49063)
0812 02:34:24.638776 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:24.638902 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":314,"threads_started":1}
I20260812 02:34:24.651531 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:24.650088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:43159)
0812 02:34:24.650956 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:24.651183 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":353,"threads_started":1}
I20260812 02:34:25.152971 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:25.152098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:59523)
0812 02:34:25.152624 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:25.152740 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":264,"threads_started":1}
I20260812 02:34:25.641402 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:25.640420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:56735)
0812 02:34:25.640995 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:25.641124 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":304,"threads_started":1}
I20260812 02:34:25.654266 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:25.652966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:55401)
0812 02:34:25.653901 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:25.654014 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":669,"threads_started":1}
I20260812 02:34:26.155272 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:26.154236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:47063)
0812 02:34:26.154920 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:26.155038 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":341,"threads_started":1}
I20260812 02:34:26.643567 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:26.642657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:52277)
0812 02:34:26.643245 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:26.643358 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":350,"threads_started":1}
I20260812 02:34:26.656481 795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:26.655391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:43701)
0812 02:34:26.656144 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:26.656242 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":459,"threads_started":1}
I20260812 02:34:27.157341 796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:27.156430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:40293)
0812 02:34:27.156994 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:27.157098 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":318,"threads_started":1}
I20260812 02:34:27.645905 797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:27.644866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:50347)
0812 02:34:27.645507 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:27.645650 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":373,"threads_started":1}
I20260812 02:34:27.658767 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:27.657835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:44843)
0812 02:34:27.658423 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:27.658523 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":336,"threads_started":1}
I20260812 02:34:28.160135 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:28.158527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:51207)
0812 02:34:28.159733 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:28.159861 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":399,"threads_started":1}
I20260812 02:34:28.647915 800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:28.647033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:34561)
0812 02:34:28.647603 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:28.647712 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":296,"threads_started":1}
I20260812 02:34:28.660916 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:28.659949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:45445)
0812 02:34:28.660546 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:28.660661 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":284,"threads_started":1}
I20260812 02:34:29.162602 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:29.161466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:39259)
0812 02:34:29.162207 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:29.162345 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":328,"threads_started":1}
I20260812 02:34:29.650226 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:29.649036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:42267)
0812 02:34:29.649659 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:29.649812 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":326,"threads_started":1}
I20260812 02:34:29.662842 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:29.662025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:44759)
0812 02:34:29.662544 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:29.662635 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
I20260812 02:34:30.164944 807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:30.163980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:53771)
0812 02:34:30.164578 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:30.164704 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":293,"threads_started":1}
I20260812 02:34:30.652407 808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:30.651420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:34847)
0812 02:34:30.652075 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:30.652170 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":356,"threads_started":1}
I20260812 02:34:30.664798 809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:30.663867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:37261)
0812 02:34:30.664443 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:30.664568 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":278,"threads_started":1}
I20260812 02:34:31.167002 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:31.166075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:56227)
0812 02:34:31.166643 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:31.166759 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":252,"threads_started":1}
I20260812 02:34:31.655022 812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:31.654018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:35853)
0812 02:34:31.654603 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:31.654749 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":302,"threads_started":1}
I20260812 02:34:31.667152 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:31.666023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:39441)
0812 02:34:31.666726 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:31.666851 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":309,"threads_started":1}
I20260812 02:34:32.169293 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:32.168389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.194:48481)
0812 02:34:32.168902 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:32.169019 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":250,"threads_started":1}
I20260812 02:34:32.630510 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 32701
I20260812 02:34:32.657451 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:32.656185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.193:58187)
0812 02:34:32.656863 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:32.657062 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":331,"threads_started":1}
I20260812 02:34:32.658291 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:32.632022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.12.8.193:38837)
0812 02:34:32.632804 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:32.632823 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:32.632872 (+ 49us) server_negotiation.cc:407] Connection header received
0812 02:34:32.634699 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:32.634710 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:32.635292 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:32.635676 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:32.637516 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:32.639262 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:32.640943 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:32.641954 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:32.642151 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:32.649671 (+ 7520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:32.649690 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:32.649723 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:32.649818 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:32.653680 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:32.654687 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:32.654696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:32.654707 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:32.654955 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:32.655436 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:32.655441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:32.655444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:32.656305 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:32.656707 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:32.657334 (+ 627us) server_negotiation.cc:299] Negotiation successful
0812 02:34:32.657967 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":401,"threads_started":1}
I20260812 02:34:32.659909 431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:32.669724 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:32.668693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:43849 (local address 127.12.8.195:47919)
0812 02:34:32.669362 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:32.669486 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.254:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":332,"threads_started":1}
I20260812 02:34:32.753949 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 32701
I20260812 02:34:32.781564 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 442
I20260812 02:34:32.803133 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:32.782776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.12.8.194:44233)
0812 02:34:32.783438 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:32.783456 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:32.783515 (+ 59us) server_negotiation.cc:407] Connection header received
0812 02:34:32.785408 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:32.785425 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:32.786057 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:32.786460 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:32.788126 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:32.789800 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:32.791296 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:32.792238 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:32.792426 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:32.795720 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:32.795731 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:32.795753 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:32.795813 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:32.799243 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:32.800115 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:32.800123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:32.800131 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:32.800337 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:32.800687 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:32.800691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:32.800693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:32.801447 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:32.801776 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:32.802228 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 02:34:32.802758 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":305,"threads_started":1}
I20260812 02:34:32.804986 593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:32.898399 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 442
I20260812 02:34:32.924055 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 602
I20260812 02:34:32.945648 822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:32.925370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.12.8.195:36489)
0812 02:34:32.926137 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:32.926154 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:32.926228 (+ 74us) server_negotiation.cc:407] Connection header received
0812 02:34:32.928127 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:32.928140 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:32.928719 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:32.929093 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:32.930786 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:32.932208 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:32.933628 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:32.934422 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:32.934580 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:32.937976 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:32.937988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:32.938011 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:32.938092 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:32.941899 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:32.942851 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:32.942859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:32.942868 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:32.943069 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:32.943411 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:32.943414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:32.943416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:32.944076 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:32.944361 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:32.944767 (+ 406us) server_negotiation.cc:299] Negotiation successful
0812 02:34:32.945294 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":398,"threads_started":1}
I20260812 02:34:32.947440 748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:33.041148 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 602
2026-08-12T02:34:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65457 ms)
[----------] 30 tests from SecurityITest (331490 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:34:33 dist-test-slave-vq2q krb5kdc[827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:34:33 dist-test-slave-vq2q krb5kdc[827](info): set up 2 sockets
Aug 12 02:34:33 dist-test-slave-vq2q krb5kdc[827](info): commencing operation
krb5kdc: starting...
W20260812 02:34:34.249013 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:34:34 dist-test-slave-vq2q krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:34:34.637337 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45527
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:45527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:35.065026 842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:35.065454 842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:35.065601 842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:35.073614 842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:35.073802 842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:35.073889 842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:35.073973 842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:35.074041 842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:35.085628 842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:45527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:35.087457 842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:35.089143 842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:35.103360 847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:35.103410 850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:35.103428 848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:35.104545 842 server_base.cc:1034] running on GCE node
I20260812 02:34:35.105712 842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:35.107666 842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:35.108964 842 hybrid_clock.cc:648] HybridClock initialized: now 1786502075108885 us; error 57 us; skew 500 ppm
I20260812 02:34:35.109485 842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:35 dist-test-slave-vq2q krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502075, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:35.113437 842 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:34:35.115578 842 webserver.cc:460] Webserver started at http://127.12.8.254:33205/ using document root <none> and password file <none>
I20260812 02:34:35.116496 842 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:35.116638 842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:35.117007 842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:35.120821 842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "b889ff31c6624290a6fefd178d16ada9"
format_stamp: "Formatted at 2026-08-12 02:34:35 on dist-test-slave-vq2q"
server_key: "0e8fc4bab88f807d8333979e259e9c3f"
server_key_iv: "abdaa29a5efba2dff81a3e9d5a2e9caf"
server_key_version: "encryptionkey@0"
I20260812 02:34:35.121745 842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "b889ff31c6624290a6fefd178d16ada9"
format_stamp: "Formatted at 2026-08-12 02:34:35 on dist-test-slave-vq2q"
server_key: "0e8fc4bab88f807d8333979e259e9c3f"
server_key_iv: "abdaa29a5efba2dff81a3e9d5a2e9caf"
server_key_version: "encryptionkey@0"
I20260812 02:34:35.128491 842 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260812 02:34:35.133549 860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:35.135350 842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:35.135569 842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "b889ff31c6624290a6fefd178d16ada9"
format_stamp: "Formatted at 2026-08-12 02:34:35 on dist-test-slave-vq2q"
server_key: "0e8fc4bab88f807d8333979e259e9c3f"
server_key_iv: "abdaa29a5efba2dff81a3e9d5a2e9caf"
server_key_version: "encryptionkey@0"
I20260812 02:34:35.135807 842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:35.165999 842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:35.171242 842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:35.171594 842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:35.193295 842 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:45527
I20260812 02:34:35.193377 919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:45527 every 8 connection(s)
I20260812 02:34:35.194816 842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:35.197896 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 842
I20260812 02:34:35.198130 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:34:35.198889 12323 external_mini_cluster.cc:1428] Setting key 24a5ee9092a5aa57a919bdb40fb4b615
I20260812 02:34:35.199852 920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 02:34:35 dist-test-slave-vq2q krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:35.217566 920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9: Bootstrap starting.
I20260812 02:34:35.223016 920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:35.224572 920 log.cc:824] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:35.229001 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:35.201270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.12.8.254:45527)
0812 02:34:35.202295 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:35.202315 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:35.202380 (+ 65us) server_negotiation.cc:407] Connection header received
0812 02:34:35.204401 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:35.204436 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:35.205153 (+ 717us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:35.205776 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:35.207292 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:35.209547 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:35.211211 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:35.212410 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:35.212710 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:35.218332 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:35.218346 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:35.218372 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:35.218434 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:35.223030 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:35.224147 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:35.224162 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:35.224175 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:35.224464 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:35.224993 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:35.224998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:35.225001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:35.225930 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:35.226226 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:35.226872 (+ 646us) server_negotiation.cc:299] Negotiation successful
0812 02:34:35.227436 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":383,"threads_started":1}
I20260812 02:34:35.229250 920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9: No bootstrap required, opened a new log
I20260812 02:34:35.238390 920 raft_consensus.cc:348] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889ff31c6624290a6fefd178d16ada9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45527 } }
I20260812 02:34:35.239053 920 raft_consensus.cc:374] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:35.239233 920 raft_consensus.cc:729] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b889ff31c6624290a6fefd178d16ada9, State: Initialized, Role: FOLLOWER
I20260812 02:34:35.240120 920 consensus_queue.cc:260] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889ff31c6624290a6fefd178d16ada9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45527 } }
I20260812 02:34:35.240435 920 raft_consensus.cc:388] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:35.240567 920 raft_consensus.cc:482] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:35.240728 920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:35.243450 920 raft_consensus.cc:504] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889ff31c6624290a6fefd178d16ada9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45527 } }
I20260812 02:34:35.244150 920 leader_election.cc:302] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b889ff31c6624290a6fefd178d16ada9; no voters:
I20260812 02:34:35.244927 920 leader_election.cc:288] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:35.245126 927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:35.246403 927 raft_consensus.cc:686] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [term 1 LEADER]: Becoming Leader. State: Replica: b889ff31c6624290a6fefd178d16ada9, State: Running, Role: LEADER
I20260812 02:34:35.247232 927 consensus_queue.cc:237] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889ff31c6624290a6fefd178d16ada9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45527 } }
I20260812 02:34:35.247695 920 sys_catalog.cc:564] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:35.250586 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b889ff31c6624290a6fefd178d16ada9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889ff31c6624290a6fefd178d16ada9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45527 } } }
I20260812 02:34:35.250499 929 sys_catalog.cc:455] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b889ff31c6624290a6fefd178d16ada9. Latest consensus state: current_term: 1 leader_uuid: "b889ff31c6624290a6fefd178d16ada9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b889ff31c6624290a6fefd178d16ada9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 45527 } } }
I20260812 02:34:35.251034 929 sys_catalog.cc:458] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:35.251037 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:35.252151 936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:35.258767 936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:35.268368 936 catalog_manager.cc:1347] Generated new cluster ID: 82bfad2810e646f1863c5f4ef9d75af8
I20260812 02:34:35.268507 936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:35.278959 936 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:35.280522 936 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:35.290478 936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9: Generated new TSK 0
I20260812 02:34:35.291428 936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:34:35.383488 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45527
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:35.767374 951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:35.767693 951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:35.767840 951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:35.775491 951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:35.775631 951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:35.775815 951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:34:35.787142 951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:35.788516 951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:35.789984 951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:35.804612 959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:35.804719 958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:35.804682 961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:35.805733 951 server_base.cc:1034] running on GCE node
I20260812 02:34:35.806535 951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:35.807732 951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:35.809000 951 hybrid_clock.cc:648] HybridClock initialized: now 1786502075808908 us; error 59 us; skew 500 ppm
I20260812 02:34:35.809481 951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:35 dist-test-slave-vq2q krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502075, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:35.813561 951 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:34:35.815653 951 webserver.cc:460] Webserver started at http://127.12.8.193:36757/ using document root <none> and password file <none>
I20260812 02:34:35.816582 951 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:35.816720 951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:35.817121 951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:35.820904 951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "21c6fe77904f46a6a484b95996e9dab0"
format_stamp: "Formatted at 2026-08-12 02:34:35 on dist-test-slave-vq2q"
server_key: "76cacc447d04816eca6c4d7c0093986e"
server_key_iv: "3d936352bb3b51a35775012c5ebcdbbc"
server_key_version: "encryptionkey@0"
I20260812 02:34:35.821956 951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "21c6fe77904f46a6a484b95996e9dab0"
format_stamp: "Formatted at 2026-08-12 02:34:35 on dist-test-slave-vq2q"
server_key: "76cacc447d04816eca6c4d7c0093986e"
server_key_iv: "3d936352bb3b51a35775012c5ebcdbbc"
server_key_version: "encryptionkey@0"
I20260812 02:34:35.829133 951 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 02:34:35.834108 971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:35.836042 951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 02:34:35.836277 951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "21c6fe77904f46a6a484b95996e9dab0"
format_stamp: "Formatted at 2026-08-12 02:34:35 on dist-test-slave-vq2q"
server_key: "76cacc447d04816eca6c4d7c0093986e"
server_key_iv: "3d936352bb3b51a35775012c5ebcdbbc"
server_key_version: "encryptionkey@0"
I20260812 02:34:35.836532 951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:35.858630 951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:35.864037 951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:35.864461 951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:35.865948 951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:35.867851 951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:35.867949 951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:35.868124 951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:35.868217 951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:35.908946 951 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:41475
I20260812 02:34:35.908998 1105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:41475 every 8 connection(s)
I20260812 02:34:35.910792 951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:35.915652 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 951
I20260812 02:34:35.915932 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:34:35.916805 12323 external_mini_cluster.cc:1428] Setting key 5ce0e66e572eab44e04667562ab9b244
Aug 12 02:34:35 dist-test-slave-vq2q krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502075, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:35.933145 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:35.914045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:58843 (local address 127.12.8.254:45527)
0812 02:34:35.914644 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:35.914649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:35.915902 (+ 1253us) server_negotiation.cc:407] Connection header received
0812 02:34:35.918227 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:35.918233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:35.918338 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:35.918437 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:35.920348 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:35.921130 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:35.922812 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:35.923227 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:35.923361 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:35.927266 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:35.927272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:35.927276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:35.927312 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:35.929934 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:35.930726 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:35.930732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:35.930734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:35.930861 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:35.931298 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:35.931303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:35.931306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:35.931605 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:35.931744 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:35.932524 (+ 780us) server_negotiation.cc:299] Negotiation successful
0812 02:34:35.932773 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":276,"threads_started":1}
I20260812 02:34:35.934254 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:35.914712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45527 (local address 127.12.8.193:58843)
0812 02:34:35.915631 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:35.915694 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:35.917925 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:35.918728 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:35.918737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:35.919484 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:35.920150 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:35.920169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:35.921316 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:35.921321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:35.922645 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:35.922659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:35.923496 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:35.923503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:35.923748 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:35.924453 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:35.924484 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:35.927003 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:35.930112 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:35.930123 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:35.930139 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:35.930602 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:35.930969 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:35.930974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:35.930978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:35.931188 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:35.931735 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:35.931743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:35.932129 (+ 386us) client_negotiation.cc:769] Sending connection context
0812 02:34:35.932403 (+ 274us) client_negotiation.cc:240] Negotiation successful
0812 02:34:35.932995 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":400,"threads_started":1}
I20260812 02:34:35.936582 1106 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45527
I20260812 02:34:35.936955 1106 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:35.937789 1106 heartbeater.cc:511] Master 127.12.8.254:45527 requested a full tablet report, sending...
I20260812 02:34:35.940798 879 ts_manager.cc:194] Registered new tserver with Master: 21c6fe77904f46a6a484b95996e9dab0 (127.12.8.193:41475)
I20260812 02:34:35.942775 879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:58843
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:34:36.000783 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45527
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:36.359851 1115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:36.360168 1115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:36.360296 1115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:36.367543 1115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:36.367674 1115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:36.367830 1115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:34:36.377633 1115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:36.378970 1115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:36.380329 1115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:36.395318 1121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:36.395500 1123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:36.395504 1120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:36.396561 1115 server_base.cc:1034] running on GCE node
I20260812 02:34:36.397256 1115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:36.398319 1115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:36.399559 1115 hybrid_clock.cc:648] HybridClock initialized: now 1786502076399483 us; error 69 us; skew 500 ppm
I20260812 02:34:36.399940 1115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:36 dist-test-slave-vq2q krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502076, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:36.403486 1115 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:34:36.405341 1115 webserver.cc:460] Webserver started at http://127.12.8.194:37321/ using document root <none> and password file <none>
I20260812 02:34:36.406107 1115 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:36.406243 1115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:36.406575 1115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:36.409747 1115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "3069506dbc17484b801ef09e19965ad8"
format_stamp: "Formatted at 2026-08-12 02:34:36 on dist-test-slave-vq2q"
server_key: "59826fdf23b8de64099f0c2ea0a3a016"
server_key_iv: "12de43d1585fb203071c5249dc9afcce"
server_key_version: "encryptionkey@0"
I20260812 02:34:36.410637 1115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "3069506dbc17484b801ef09e19965ad8"
format_stamp: "Formatted at 2026-08-12 02:34:36 on dist-test-slave-vq2q"
server_key: "59826fdf23b8de64099f0c2ea0a3a016"
server_key_iv: "12de43d1585fb203071c5249dc9afcce"
server_key_version: "encryptionkey@0"
I20260812 02:34:36.416621 1115 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 02:34:36.421005 1131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:36.422628 1115 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 02:34:36.422786 1115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "3069506dbc17484b801ef09e19965ad8"
format_stamp: "Formatted at 2026-08-12 02:34:36 on dist-test-slave-vq2q"
server_key: "59826fdf23b8de64099f0c2ea0a3a016"
server_key_iv: "12de43d1585fb203071c5249dc9afcce"
server_key_version: "encryptionkey@0"
I20260812 02:34:36.422994 1115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:36.435755 1115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:36.440201 1115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:36.440503 1115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:36.441706 1115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:36.443259 1115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:36.443341 1115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:36.443446 1115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:36.443499 1115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:36.480073 1115 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:46525
I20260812 02:34:36.480192 1259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:46525 every 8 connection(s)
I20260812 02:34:36.481478 1115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:36.482136 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 1115
I20260812 02:34:36.482345 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:34:36.483139 12323 external_mini_cluster.cc:1428] Setting key 73a845f50992f44e23b526048a898a3c
Aug 12 02:34:36 dist-test-slave-vq2q krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502076, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:36.506665 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:36.484585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:52735 (local address 127.12.8.254:45527)
0812 02:34:36.485221 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:36.485227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:36.486248 (+ 1021us) server_negotiation.cc:407] Connection header received
0812 02:34:36.488422 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:36.488427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:36.488518 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:36.488624 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:36.490582 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:36.491410 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:36.493135 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:36.493674 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:36.493794 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:36.498192 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:36.498199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:36.498202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:36.498248 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:36.501964 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:36.503233 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:36.503240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:36.503243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:36.503543 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:36.504251 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:36.504256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:36.504258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:36.504623 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:36.504761 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:36.506103 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0812 02:34:36.506369 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":279,"threads_started":1}
I20260812 02:34:36.508534 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:36.485103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45527 (local address 127.12.8.194:52735)
0812 02:34:36.486058 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:36.486110 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:36.488125 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:36.488918 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:36.488927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:36.489651 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:36.490378 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:36.490402 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:36.491586 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:36.491590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:36.492973 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:36.492997 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:36.493933 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:36.493940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:36.494224 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:36.494990 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:36.495019 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:36.497842 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:36.502190 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:36.502203 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:36.502225 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:36.502994 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:36.503711 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:36.503718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:36.503722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:36.504098 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:36.504782 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:36.504792 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:36.505527 (+ 735us) client_negotiation.cc:769] Sending connection context
0812 02:34:36.505937 (+ 410us) client_negotiation.cc:240] Negotiation successful
0812 02:34:36.506660 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":374,"threads_started":1}
I20260812 02:34:36.510643 1260 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45527
I20260812 02:34:36.511008 1260 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:36.511888 1260 heartbeater.cc:511] Master 127.12.8.254:45527 requested a full tablet report, sending...
I20260812 02:34:36.513767 879 ts_manager.cc:194] Registered new tserver with Master: 3069506dbc17484b801ef09e19965ad8 (127.12.8.194:46525)
I20260812 02:34:36.515161 879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:52735
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:34:36.569522 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45527
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:36.947273 1106 heartbeater.cc:503] Master 127.12.8.254:45527 was elected leader, sending a full tablet report...
W20260812 02:34:37.005808 1268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:37.006255 1268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:37.006402 1268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:37.014863 1268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:37.015024 1268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:37.015200 1268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:34:37.027526 1268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:45527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:37.029388 1268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:37.031083 1268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:37.047223 1274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:37.047266 1276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:37.047406 1273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:37.049153 1268 server_base.cc:1034] running on GCE node
I20260812 02:34:37.050017 1268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:37.051263 1268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:37.052547 1268 hybrid_clock.cc:648] HybridClock initialized: now 1786502077052479 us; error 45 us; skew 500 ppm
I20260812 02:34:37.052981 1268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:37 dist-test-slave-vq2q krb5kdc[827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502077, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:37.057271 1268 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:34:37.059147 1268 webserver.cc:460] Webserver started at http://127.12.8.195:43219/ using document root <none> and password file <none>
I20260812 02:34:37.060060 1268 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:37.060194 1268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:37.060581 1268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:37.064270 1268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "4fa72e7f1c9d48708bbc32719fbdc852"
format_stamp: "Formatted at 2026-08-12 02:34:37 on dist-test-slave-vq2q"
server_key: "e093d5c14a81483eea76e135fb064dfa"
server_key_iv: "cea0c496e503ece01974a826c7e97c46"
server_key_version: "encryptionkey@0"
I20260812 02:34:37.065250 1268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "4fa72e7f1c9d48708bbc32719fbdc852"
format_stamp: "Formatted at 2026-08-12 02:34:37 on dist-test-slave-vq2q"
server_key: "e093d5c14a81483eea76e135fb064dfa"
server_key_iv: "cea0c496e503ece01974a826c7e97c46"
server_key_version: "encryptionkey@0"
I20260812 02:34:37.072391 1268 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260812 02:34:37.076838 1285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:37.078723 1268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:37.078944 1268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "4fa72e7f1c9d48708bbc32719fbdc852"
format_stamp: "Formatted at 2026-08-12 02:34:37 on dist-test-slave-vq2q"
server_key: "e093d5c14a81483eea76e135fb064dfa"
server_key_iv: "cea0c496e503ece01974a826c7e97c46"
server_key_version: "encryptionkey@0"
I20260812 02:34:37.079197 1268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:37.103296 1268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:37.108417 1268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:37.108768 1268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:37.110111 1268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:37.111801 1268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:37.111888 1268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:37.112018 1268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:37.112074 1268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:37.149916 1268 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:37625
I20260812 02:34:37.150022 1427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:37625 every 8 connection(s)
I20260812 02:34:37.151597 1268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:37.160809 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 1268
I20260812 02:34:37.161028 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:34:37.161662 12323 external_mini_cluster.cc:1428] Setting key cab9ffeb60ab6214c05ccb1fd12c67d0
Aug 12 02:34:37 dist-test-slave-vq2q krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502077, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:37.176632 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.154893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:50469 (local address 127.12.8.254:45527)
0812 02:34:37.155572 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.155580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.156747 (+ 1167us) server_negotiation.cc:407] Connection header received
0812 02:34:37.159046 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.159051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.159158 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.159291 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:37.161511 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.162517 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.164618 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.165090 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.165265 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.169462 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:37.169469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:37.169473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:37.169521 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:37.172507 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.173712 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.173717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.173720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.173865 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.174375 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.174379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.174381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.174690 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:37.174866 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.175958 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.176249 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":286,"threads_started":1}
I20260812 02:34:37.178333 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.155501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:45527 (local address 127.12.8.195:50469)
0812 02:34:37.156513 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:37.156570 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:37.158646 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:37.159583 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:37.159592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:37.160259 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:37.161172 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.161236 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.162718 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.162724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.164436 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.164454 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.165402 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.165410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.165663 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.166427 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:37.166457 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:37.169130 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:37.172737 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:37.172748 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:37.172767 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:37.173542 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:37.173968 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:37.173974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:37.173977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:37.174235 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:37.174885 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:37.174893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:37.175423 (+ 530us) client_negotiation.cc:769] Sending connection context
0812 02:34:37.175792 (+ 369us) client_negotiation.cc:240] Negotiation successful
0812 02:34:37.176492 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":380,"threads_started":1}
I20260812 02:34:37.180459 1429 heartbeater.cc:347] Connected to a master server at 127.12.8.254:45527
I20260812 02:34:37.180855 1429 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:37.181830 1429 heartbeater.cc:511] Master 127.12.8.254:45527 requested a full tablet report, sending...
I20260812 02:34:37.183660 879 ts_manager.cc:194] Registered new tserver with Master: 4fa72e7f1c9d48708bbc32719fbdc852 (127.12.8.195:37625)
I20260812 02:34:37.184837 879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:50469
I20260812 02:34:37.192613 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:37.214226 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.198487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.12.8.254:45527)
0812 02:34:37.198724 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.198732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.199168 (+ 436us) server_negotiation.cc:407] Connection header received
0812 02:34:37.199388 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.199393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.199473 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.199586 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:37.201666 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.202683 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.204370 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.204873 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.205006 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.208003 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:37.208009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:37.208012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:37.208060 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:37.211092 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.211917 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.211922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.211925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.212069 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.212463 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.212468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.212470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.212897 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:37.213157 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.213571 (+ 414us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.213853 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:34:37.219696 879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:37.229398 879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:37.257769 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.245607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:41475 (local address 127.0.0.1:46294)
0812 02:34:37.247256 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:37.247311 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:37.247619 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:37.248781 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:37.248789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:37.248825 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:37.249480 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.249496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.251546 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.251552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.254156 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.254170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.256566 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.256573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.256808 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.256868 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 02:34:37.257027 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 02:34:37.257301 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"thread_start_us":3418,"threads_started":1}
I20260812 02:34:37.257817 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.245788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.12.8.193:41475)
0812 02:34:37.246579 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.246592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.247435 (+ 843us) server_negotiation.cc:407] Connection header received
0812 02:34:37.247910 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.247921 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.248234 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.248418 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:37.249735 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.251332 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.254375 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.255912 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.256070 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.256369 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.257146 (+ 777us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.257447 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":672,"threads_started":1}
I20260812 02:34:37.258885 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.246270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.12.8.195:37625)
0812 02:34:37.248712 (+ 2442us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.248719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.249554 (+ 835us) server_negotiation.cc:407] Connection header received
0812 02:34:37.249937 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.249947 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.250157 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.250362 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:37.251516 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.253154 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.255940 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.257465 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.257575 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.257835 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.258401 (+ 566us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.258560 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2269,"thread_start_us":2080,"threads_started":1}
I20260812 02:34:37.258872 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.245771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:37625 (local address 127.0.0.1:54436)
0812 02:34:37.249406 (+ 3635us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:37.249429 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:37.249687 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:37.250684 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:37.250690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:37.250716 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:37.251315 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.251326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.253320 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.253326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.255695 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.255712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.258040 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.258046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.258201 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.258231 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:34:37.258320 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 02:34:37.258471 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3509,"thread_start_us":998,"threads_started":1}
I20260812 02:34:37.262101 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.250659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:46525 (local address 127.0.0.1:49738)
0812 02:34:37.251381 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:37.251407 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:37.251635 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:37.252708 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:37.252712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:37.252744 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:37.253248 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.253262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.255276 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.255281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.257552 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.257568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.261206 (+ 3638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.261213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.261407 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.261441 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 02:34:37.261557 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 02:34:37.261725 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":364,"threads_started":1}
I20260812 02:34:37.262305 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.250033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.12.8.194:46525)
0812 02:34:37.250677 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.250686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.251518 (+ 832us) server_negotiation.cc:407] Connection header received
0812 02:34:37.251868 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.251878 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.252156 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.252356 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:37.253485 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.255096 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.257799 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.260584 (+ 2785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.260719 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.260950 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.261682 (+ 732us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.261991 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":298,"threads_started":1}
I20260812 02:34:37.263726 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 22b4f7a60bbc4a3f9cc103d213ed5242 (DEFAULT_TABLE table=test-table [id=b6d53a9793824710b340b8195f7d7586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:37.264043 1346 tablet_service.cc:1467] Processing CreateTablet for tablet 22b4f7a60bbc4a3f9cc103d213ed5242 (DEFAULT_TABLE table=test-table [id=b6d53a9793824710b340b8195f7d7586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:37.265262 1027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b4f7a60bbc4a3f9cc103d213ed5242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:37.265455 1346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b4f7a60bbc4a3f9cc103d213ed5242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:37.270569 1185 tablet_service.cc:1467] Processing CreateTablet for tablet 22b4f7a60bbc4a3f9cc103d213ed5242 (DEFAULT_TABLE table=test-table [id=b6d53a9793824710b340b8195f7d7586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:37.273125 1185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b4f7a60bbc4a3f9cc103d213ed5242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:37.278636 1454 tablet_bootstrap.cc:492] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Bootstrap starting.
I20260812 02:34:37.282686 1454 tablet_bootstrap.cc:654] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:37.282912 1455 tablet_bootstrap.cc:492] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Bootstrap starting.
I20260812 02:34:37.284345 1454 log.cc:824] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:37.287223 1455 tablet_bootstrap.cc:654] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:37.287752 1454 tablet_bootstrap.cc:492] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: No bootstrap required, opened a new log
I20260812 02:34:37.288018 1454 ts_tablet_manager.cc:1397] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260812 02:34:37.288697 1455 log.cc:824] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:37.291232 1458 tablet_bootstrap.cc:492] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: Bootstrap starting.
I20260812 02:34:37.292738 1455 tablet_bootstrap.cc:492] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: No bootstrap required, opened a new log
I20260812 02:34:37.293094 1455 ts_tablet_manager.cc:1397] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260812 02:34:37.295496 1458 tablet_bootstrap.cc:654] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:37.296492 1454 raft_consensus.cc:348] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.296875 1454 raft_consensus.cc:374] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:37.296974 1454 raft_consensus.cc:729] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa72e7f1c9d48708bbc32719fbdc852, State: Initialized, Role: FOLLOWER
I20260812 02:34:37.296984 1458 log.cc:824] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:37.297616 1454 consensus_queue.cc:260] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.299057 1429 heartbeater.cc:503] Master 127.12.8.254:45527 was elected leader, sending a full tablet report...
I20260812 02:34:37.299530 1454 ts_tablet_manager.cc:1428] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260812 02:34:37.303025 1458 tablet_bootstrap.cc:492] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: No bootstrap required, opened a new log
I20260812 02:34:37.303390 1458 ts_tablet_manager.cc:1397] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260812 02:34:37.304076 1455 raft_consensus.cc:348] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.304628 1455 raft_consensus.cc:374] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:37.304751 1455 raft_consensus.cc:729] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c6fe77904f46a6a484b95996e9dab0, State: Initialized, Role: FOLLOWER
I20260812 02:34:37.305685 1455 consensus_queue.cc:260] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.309396 1455 ts_tablet_manager.cc:1428] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260812 02:34:37.313382 1458 raft_consensus.cc:348] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.313841 1458 raft_consensus.cc:374] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:37.313946 1458 raft_consensus.cc:729] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3069506dbc17484b801ef09e19965ad8, State: Initialized, Role: FOLLOWER
I20260812 02:34:37.314630 1458 consensus_queue.cc:260] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.315968 1260 heartbeater.cc:503] Master 127.12.8.254:45527 was elected leader, sending a full tablet report...
I20260812 02:34:37.316695 1458 ts_tablet_manager.cc:1428] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260812 02:34:37.359438 1462 raft_consensus.cc:482] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:37.359862 1462 raft_consensus.cc:504] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.361785 1462 leader_election.cc:288] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21c6fe77904f46a6a484b95996e9dab0 (127.12.8.193:41475), 4fa72e7f1c9d48708bbc32719fbdc852 (127.12.8.195:37625)
I20260812 02:34:37.369982 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.362493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:47155 (local address 127.12.8.195:37625)
0812 02:34:37.362747 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.362755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.363635 (+ 880us) server_negotiation.cc:407] Connection header received
0812 02:34:37.363848 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.363855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.363942 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.364030 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:37.364907 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.365780 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.368087 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.368873 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.369016 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.369075 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.369568 (+ 493us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.369711 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 02:34:37.370076 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.362513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:56091 (local address 127.12.8.193:41475)
0812 02:34:37.362778 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.362786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.363756 (+ 970us) server_negotiation.cc:407] Connection header received
0812 02:34:37.364095 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.364101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.364208 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.364338 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:37.365123 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.366023 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.368087 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.368991 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.369209 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.369299 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.369646 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.369800 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:34:37.370361 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.362398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:41475 (local address 127.12.8.194:56091)
0812 02:34:37.363576 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:37.363607 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:37.363801 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:37.364379 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:37.364383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:37.364408 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:37.364916 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.364930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.366207 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.366211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.367889 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.367903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.369290 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.369295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.369453 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.369484 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:34:37.369593 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 02:34:37.369750 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":1,"thread_start_us":646,"threads_started":1}
I20260812 02:34:37.370674 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.362147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:37625 (local address 127.12.8.194:47155)
0812 02:34:37.363463 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:37.363486 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:37.363673 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:37.364127 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:37.364132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:37.364173 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:37.364727 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.364740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.365936 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.365942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.367902 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:37.367919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.369150 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.369158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:37.369387 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.369418 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:34:37.369505 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 02:34:37.369668 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":80,"thread_start_us":896,"threads_started":1}
I20260812 02:34:37.371742 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b4f7a60bbc4a3f9cc103d213ed5242" candidate_uuid: "3069506dbc17484b801ef09e19965ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c6fe77904f46a6a484b95996e9dab0" is_pre_election: true
I20260812 02:34:37.371946 1372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b4f7a60bbc4a3f9cc103d213ed5242" candidate_uuid: "3069506dbc17484b801ef09e19965ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" is_pre_election: true
I20260812 02:34:37.372401 1049 raft_consensus.cc:2454] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3069506dbc17484b801ef09e19965ad8 in term 0.
I20260812 02:34:37.372510 1372 raft_consensus.cc:2454] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3069506dbc17484b801ef09e19965ad8 in term 0.
I20260812 02:34:37.373309 1132 leader_election.cc:302] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3069506dbc17484b801ef09e19965ad8, 4fa72e7f1c9d48708bbc32719fbdc852; no voters:
I20260812 02:34:37.373746 1462 raft_consensus.cc:2781] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:37.373916 1462 raft_consensus.cc:482] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:37.374024 1462 raft_consensus.cc:3037] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:37.376444 1462 raft_consensus.cc:504] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.377115 1462 leader_election.cc:288] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [CANDIDATE]: Term 1 election: Requested vote from peers 21c6fe77904f46a6a484b95996e9dab0 (127.12.8.193:41475), 4fa72e7f1c9d48708bbc32719fbdc852 (127.12.8.195:37625)
I20260812 02:34:37.377770 1372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b4f7a60bbc4a3f9cc103d213ed5242" candidate_uuid: "3069506dbc17484b801ef09e19965ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa72e7f1c9d48708bbc32719fbdc852"
I20260812 02:34:37.377832 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b4f7a60bbc4a3f9cc103d213ed5242" candidate_uuid: "3069506dbc17484b801ef09e19965ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c6fe77904f46a6a484b95996e9dab0"
I20260812 02:34:37.377972 1372 raft_consensus.cc:3037] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:37.378139 1049 raft_consensus.cc:3037] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:37.380793 1372 raft_consensus.cc:2454] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3069506dbc17484b801ef09e19965ad8 in term 1.
I20260812 02:34:37.381313 1132 leader_election.cc:302] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3069506dbc17484b801ef09e19965ad8, 4fa72e7f1c9d48708bbc32719fbdc852; no voters:
I20260812 02:34:37.381409 1049 raft_consensus.cc:2454] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3069506dbc17484b801ef09e19965ad8 in term 1.
I20260812 02:34:37.381698 1462 raft_consensus.cc:2781] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:37.382321 1462 raft_consensus.cc:686] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 3069506dbc17484b801ef09e19965ad8, State: Running, Role: LEADER
I20260812 02:34:37.382995 1462 consensus_queue.cc:237] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } }
I20260812 02:34:37.388861 877 catalog_manager.cc:5495] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3069506dbc17484b801ef09e19965ad8 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "3069506dbc17484b801ef09e19965ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3069506dbc17484b801ef09e19965ad8" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46525 } health_report { overall_health: HEALTHY } } }
W20260812 02:34:37.403542 1430 tablet.cc:2369] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:34:37.414144 1107 tablet.cc:2369] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:37.426669 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.419529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.12.8.194:46525)
0812 02:34:37.419811 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.419817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.419842 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:34:37.420246 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.420251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.420379 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.420590 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:37.421126 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.422037 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.423744 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.424228 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.424370 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.425084 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:37.425861 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:37.426024 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.426153 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.426298 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 02:34:37.437637 1372 raft_consensus.cc:1261] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 1 FOLLOWER]: Refusing update from remote peer 3069506dbc17484b801ef09e19965ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:37.437783 1049 raft_consensus.cc:1261] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 1 FOLLOWER]: Refusing update from remote peer 3069506dbc17484b801ef09e19965ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:37.438853 1462 consensus_queue.cc:1035] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa72e7f1c9d48708bbc32719fbdc852" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:37.439213 1465 consensus_queue.cc:1035] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c6fe77904f46a6a484b95996e9dab0" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 41475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:37.459642 1468 mvcc.cc:205] Tried to move back new op lower bound from 7317512509171081216 to 7317512508965720064. Current Snapshot: MvccSnapshot[applied={T|T < 7317512509171081216}]
I20260812 02:34:37.472237 1472 mvcc.cc:205] Tried to move back new op lower bound from 7317512509171081216 to 7317512508965720064. Current Snapshot: MvccSnapshot[applied={T|T < 7317512509171081216}]
I20260812 02:34:37.486186 877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:37.486541 877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:37.496062 877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b889ff31c6624290a6fefd178d16ada9: Sending DeleteTablet for 3 replicas of tablet 22b4f7a60bbc4a3f9cc103d213ed5242
I20260812 02:34:37.497547 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b4f7a60bbc4a3f9cc103d213ed5242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:37 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:46294
I20260812 02:34:37.498349 1346 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b4f7a60bbc4a3f9cc103d213ed5242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:37 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:54436
I20260812 02:34:37.498701 1483 tablet_replica.cc:330] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: stopping tablet replica
I20260812 02:34:37.499332 1483 raft_consensus.cc:2229] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:37.500005 1483 raft_consensus.cc:2258] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:37.501880 1185 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b4f7a60bbc4a3f9cc103d213ed5242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:37 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:49738
I20260812 02:34:37.502559 1345 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b4f7a60bbc4a3f9cc103d213ed5242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:37 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:54436
I20260812 02:34:37.503086 1485 tablet_replica.cc:330] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: stopping tablet replica
I20260812 02:34:37.503717 1485 raft_consensus.cc:2229] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:34:37.505893 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 951
I20260812 02:34:37.510582 1483 ts_tablet_manager.cc:1905] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:37.517783 1485 raft_consensus.cc:2258] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:37.520970 1485 ts_tablet_manager.cc:1905] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:37.527060 1484 tablet_replica.cc:330] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: stopping tablet replica
I20260812 02:34:37.527629 1484 raft_consensus.cc:2229] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:37.528227 1484 raft_consensus.cc:2258] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:37.530746 1485 ts_tablet_manager.cc:1918] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:37.531032 1485 log.cc:1196] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/22b4f7a60bbc4a3f9cc103d213ed5242
I20260812 02:34:37.531756 1485 ts_tablet_manager.cc:1939] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 3069506dbc17484b801ef09e19965ad8: Deleting consensus metadata
I20260812 02:34:37.531736 1483 ts_tablet_manager.cc:1918] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:37.531971 1483 log.cc:1196] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/22b4f7a60bbc4a3f9cc103d213ed5242
I20260812 02:34:37.532548 1483 ts_tablet_manager.cc:1939] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 21c6fe77904f46a6a484b95996e9dab0: Deleting consensus metadata
I20260812 02:34:37.534296 864 catalog_manager.cc:4841] TS 3069506dbc17484b801ef09e19965ad8 (127.12.8.194:46525): tablet 22b4f7a60bbc4a3f9cc103d213ed5242 (table test-table [id=b6d53a9793824710b340b8195f7d7586]) successfully deleted
I20260812 02:34:37.534721 864 catalog_manager.cc:4841] TS 21c6fe77904f46a6a484b95996e9dab0 (127.12.8.193:41475): tablet 22b4f7a60bbc4a3f9cc103d213ed5242 (table test-table [id=b6d53a9793824710b340b8195f7d7586]) successfully deleted
I20260812 02:34:37.537139 1484 ts_tablet_manager.cc:1905] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 02:34:37 dist-test-slave-vq2q krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:34:37.545693 1484 ts_tablet_manager.cc:1918] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:37.545928 1484 log.cc:1196] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/22b4f7a60bbc4a3f9cc103d213ed5242
I20260812 02:34:37.546507 1484 ts_tablet_manager.cc:1939] T 22b4f7a60bbc4a3f9cc103d213ed5242 P 4fa72e7f1c9d48708bbc32719fbdc852: Deleting consensus metadata
I20260812 02:34:37.548653 861 catalog_manager.cc:4841] TS 4fa72e7f1c9d48708bbc32719fbdc852 (127.12.8.195:37625): tablet 22b4f7a60bbc4a3f9cc103d213ed5242 (table test-table [id=b6d53a9793824710b340b8195f7d7586]) successfully deleted
W20260812 02:34:37.549741 861 catalog_manager.cc:4808] TS 4fa72e7f1c9d48708bbc32719fbdc852 (127.12.8.195:37625): delete failed for tablet 22b4f7a60bbc4a3f9cc103d213ed5242 because the tablet was not found. No further retry: Not found: Tablet not found: 22b4f7a60bbc4a3f9cc103d213ed5242
I20260812 02:34:37.550841 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.507887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.12.8.193:41475)
0812 02:34:37.508109 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.508115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.526166 (+ 18051us) server_negotiation.cc:407] Connection header received
0812 02:34:37.526374 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.526381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.526508 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.526636 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:37.529369 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.530576 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.533080 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.533801 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.533907 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.540292 (+ 6385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:37.540317 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:37.540344 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:37.540399 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:37.544894 (+ 4495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.546447 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.546462 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.546466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.546822 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.547371 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.547376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.547379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.548035 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:37.549110 (+ 1075us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.550077 (+ 967us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.550408 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 02:34:37.552690 1101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:37.670838 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 951
I20260812 02:34:37.697893 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 1115
Aug 12 02:34:37 dist-test-slave-vq2q krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:34:37.723394 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.698810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.12.8.194:46525)
0812 02:34:37.699038 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.699044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.699070 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:34:37.699262 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.699267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.699361 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.699502 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:37.700921 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.702158 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.703951 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.704619 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.704758 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.715421 (+ 10663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:37.715437 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:37.715458 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:37.715517 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:37.719505 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.720607 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.720616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.720620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.720840 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.721571 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.721577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.721581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.722063 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:37.722413 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.722739 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.723009 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 02:34:37.724457 1255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:37.860754 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 1115
I20260812 02:34:37.891070 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 1268
Aug 12 02:34:37 dist-test-slave-vq2q krb5kdc[827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:34:37.909384 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:37.892006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.12.8.195:37625)
0812 02:34:37.892512 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:37.892516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:37.892536 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:34:37.892599 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:37.892603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:37.892672 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:37.892802 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:37.894484 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.895218 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.896743 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:37.897145 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:37.897319 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:37.902788 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:37.902806 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:37.902822 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:37.902861 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:37.906118 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.907170 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.907178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.907180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.907358 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:37.907803 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:37.907807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:37.907809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:37.908179 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:37.908410 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:37.908846 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 02:34:37.909024 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":283,"threads_started":1}
I20260812 02:34:37.910168 1421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:38.011976 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 1268
I20260812 02:34:38.037153 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 842
I20260812 02:34:38.038738 913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:38.167716 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 842
2026-08-12T02:34:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5135 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T02:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:38Z Disabled control of system clock
I20260812 02:34:38.289513 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:37625
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:34:38.666612 1498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:38.666909 1498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:38.666997 1498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:38.674453 1498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:38.674568 1498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:38.674606 1498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:38.674660 1498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:38.674726 1498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:38.684633 1498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:37625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:38.686033 1498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:38.687913 1498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:38.701025 1509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:38.701239 1504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:38.701227 1505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:38.702592 1498 server_base.cc:1034] running on GCE node
I20260812 02:34:38.703707 1498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:38.705631 1498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:38.706869 1498 hybrid_clock.cc:648] HybridClock initialized: now 1786502078706808 us; error 53 us; skew 500 ppm
I20260812 02:34:38.707326 1498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:38.710259 1498 webserver.cc:460] Webserver started at http://127.12.8.254:41395/ using document root <none> and password file <none>
I20260812 02:34:38.711153 1498 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:38.711263 1498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:38.711658 1498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:38.715406 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "b3b086a319684a3785391690d813c186"
format_stamp: "Formatted at 2026-08-12 02:34:38 on dist-test-slave-vq2q"
server_key: "ba7b828a5f0232944ad47dcb65cf2086"
server_key_iv: "64383064bc2f0e14dd2761cf7cd56d3a"
server_key_version: "encryptionkey@0"
I20260812 02:34:38.716399 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "b3b086a319684a3785391690d813c186"
format_stamp: "Formatted at 2026-08-12 02:34:38 on dist-test-slave-vq2q"
server_key: "ba7b828a5f0232944ad47dcb65cf2086"
server_key_iv: "64383064bc2f0e14dd2761cf7cd56d3a"
server_key_version: "encryptionkey@0"
I20260812 02:34:38.723462 1498 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 02:34:38.727989 1515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:38.730057 1498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:38.730319 1498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "b3b086a319684a3785391690d813c186"
format_stamp: "Formatted at 2026-08-12 02:34:38 on dist-test-slave-vq2q"
server_key: "ba7b828a5f0232944ad47dcb65cf2086"
server_key_iv: "64383064bc2f0e14dd2761cf7cd56d3a"
server_key_version: "encryptionkey@0"
I20260812 02:34:38.730594 1498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:38.753865 1498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:38.754998 1498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:38.755342 1498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:38.777510 1498 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:37625
I20260812 02:34:38.777555 1574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:37625 every 8 connection(s)
I20260812 02:34:38.779256 1498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:38.782255 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 1498
I20260812 02:34:38.782444 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:34:38.782971 12323 external_mini_cluster.cc:1428] Setting key 9051a8a0752818be60fe57e14fe50aac
I20260812 02:34:38.783650 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:38.799590 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186: Bootstrap starting.
I20260812 02:34:38.799964 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:38.785489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.12.8.254:37625)
0812 02:34:38.786808 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:38.786826 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:38.786894 (+ 68us) server_negotiation.cc:407] Connection header received
0812 02:34:38.789022 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:38.789054 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:38.789727 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:38.790232 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:38.790890 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:38.792884 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:38.794468 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:38.795511 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:38.795741 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:38.797051 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:38.797063 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:38.797086 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:38.797863 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:38.797973 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:38.798000 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:38.798138 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:38.798324 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 02:34:38.798657 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":423,"threads_started":1}
I20260812 02:34:38.803375 1575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:38.804366 1575 log.cc:824] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:38.807410 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186: No bootstrap required, opened a new log
I20260812 02:34:38.814950 1575 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b086a319684a3785391690d813c186" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37625 } }
I20260812 02:34:38.815335 1575 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:38.815423 1575 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b086a319684a3785391690d813c186, State: Initialized, Role: FOLLOWER
I20260812 02:34:38.815992 1575 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b086a319684a3785391690d813c186" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37625 } }
I20260812 02:34:38.816268 1575 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:38.816412 1575 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:38.816578 1575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:38.818857 1575 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b086a319684a3785391690d813c186" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37625 } }
I20260812 02:34:38.819432 1575 leader_election.cc:302] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3b086a319684a3785391690d813c186; no voters:
I20260812 02:34:38.820077 1575 leader_election.cc:288] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:38.820331 1583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:38.821841 1583 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [term 1 LEADER]: Becoming Leader. State: Replica: b3b086a319684a3785391690d813c186, State: Running, Role: LEADER
I20260812 02:34:38.822396 1575 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:38.822559 1583 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b086a319684a3785391690d813c186" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37625 } }
I20260812 02:34:38.826246 1584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3b086a319684a3785391690d813c186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b086a319684a3785391690d813c186" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37625 } } }
I20260812 02:34:38.826733 1584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:38.827296 1585 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b086a319684a3785391690d813c186. Latest consensus state: current_term: 1 leader_uuid: "b3b086a319684a3785391690d813c186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b086a319684a3785391690d813c186" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 37625 } } }
I20260812 02:34:38.827646 1585 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:38.829638 1592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:38.835973 1592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:38.848127 1592 catalog_manager.cc:1347] Generated new cluster ID: 75eb8cee8224440c890b05b21d389f04
I20260812 02:34:38.848273 1592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:38.864297 1592 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:38.865702 1592 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:38.877442 1592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186: Generated new TSK 0
I20260812 02:34:38.878315 1592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:34:38.890823 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:37625
--builtin_ntp_servers=127.12.8.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:34:39.322876 1605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:39.323257 1605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:39.323392 1605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:39.331187 1605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:39.331322 1605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:39.331475 1605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:34:39.341104 1605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:39.342602 1605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:39.344617 1605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:39.359603 1612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:39.359637 1611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:39.359725 1615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:39.361009 1605 server_base.cc:1034] running on GCE node
I20260812 02:34:39.361733 1605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:39.362809 1605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:39.364032 1605 hybrid_clock.cc:648] HybridClock initialized: now 1786502079363979 us; error 61 us; skew 500 ppm
I20260812 02:34:39.364356 1605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:39.366920 1605 webserver.cc:460] Webserver started at http://127.12.8.193:40977/ using document root <none> and password file <none>
I20260812 02:34:39.367697 1605 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:39.367803 1605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:39.368132 1605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:39.371286 1605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "d6bddeec97f14e16898795b7342a56c2"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-vq2q"
server_key: "b9cdd3e0de08a5f9dc1586fc45b5f378"
server_key_iv: "f8d882a2b617eda84a9487a4ff74627b"
server_key_version: "encryptionkey@0"
I20260812 02:34:39.372102 1605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "d6bddeec97f14e16898795b7342a56c2"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-vq2q"
server_key: "b9cdd3e0de08a5f9dc1586fc45b5f378"
server_key_iv: "f8d882a2b617eda84a9487a4ff74627b"
server_key_version: "encryptionkey@0"
I20260812 02:34:39.377663 1605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 02:34:39.381637 1622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:39.383270 1605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:34:39.383459 1605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "d6bddeec97f14e16898795b7342a56c2"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-vq2q"
server_key: "b9cdd3e0de08a5f9dc1586fc45b5f378"
server_key_iv: "f8d882a2b617eda84a9487a4ff74627b"
server_key_version: "encryptionkey@0"
I20260812 02:34:39.383680 1605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:39.396255 1605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:39.397074 1605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:39.397415 1605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:39.398525 1605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:39.400199 1605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:39.400302 1605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:39.400427 1605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:39.400501 1605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:39.437813 1605 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:34773
I20260812 02:34:39.437933 1768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:34773 every 8 connection(s)
I20260812 02:34:39.439208 1605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:39.449379 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 1605
I20260812 02:34:39.449638 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:34:39.450351 12323 external_mini_cluster.cc:1428] Setting key 93e7f9caf4228fd3f63facd66f9fd952
I20260812 02:34:39.454257 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:37625
--builtin_ntp_servers=127.12.8.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:34:39.454761 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:39.442376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:56919 (local address 127.12.8.254:37625)
0812 02:34:39.442943 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:39.442949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:39.444416 (+ 1467us) server_negotiation.cc:407] Connection header received
0812 02:34:39.446538 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:39.446543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:39.446624 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:39.446741 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:39.448107 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.448821 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.451461 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.451990 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.452129 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:39.453869 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:39.453874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:39.453877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:39.453895 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:39.453939 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:39.453952 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:39.454028 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:39.454323 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 02:34:39.454422 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":259,"threads_started":1}
I20260812 02:34:39.456483 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:39.442950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:37625 (local address 127.12.8.193:56919)
0812 02:34:39.444188 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:39.444246 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:39.446243 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:39.447062 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:39.447072 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:39.447192 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:39.447928 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:39.447949 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.449070 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.449076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:39.451254 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:39.451271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.452263 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.452269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:39.452534 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:39.453419 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:39.453422 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:39.453497 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:39.453623 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:39.454065 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:39.454077 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:39.454087 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 02:34:39.454320 (+ 233us) client_negotiation.cc:240] Negotiation successful
0812 02:34:39.454821 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":353,"threads_started":1}
I20260812 02:34:39.458308 1769 heartbeater.cc:347] Connected to a master server at 127.12.8.254:37625
I20260812 02:34:39.458698 1769 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:39.459770 1769 heartbeater.cc:511] Master 127.12.8.254:37625 requested a full tablet report, sending...
I20260812 02:34:39.462422 1534 ts_manager.cc:194] Registered new tserver with Master: d6bddeec97f14e16898795b7342a56c2 (127.12.8.193:34773)
I20260812 02:34:39.463953 1534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.193:56919
W20260812 02:34:39.848850 1773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:39.849151 1773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:39.849292 1773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:39.856786 1773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:39.856896 1773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:39.857014 1773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:34:39.866340 1773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:39.867628 1773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:39.869426 1773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:39.884130 1784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:39.884366 1782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:39.884238 1781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:39.885241 1773 server_base.cc:1034] running on GCE node
I20260812 02:34:39.885972 1773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:39.887137 1773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:39.888404 1773 hybrid_clock.cc:648] HybridClock initialized: now 1786502079888296 us; error 73 us; skew 500 ppm
I20260812 02:34:39.888774 1773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:39.891356 1773 webserver.cc:460] Webserver started at http://127.12.8.194:42325/ using document root <none> and password file <none>
I20260812 02:34:39.892117 1773 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:39.892222 1773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:39.892539 1773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:39.895569 1773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "883fb74dd4bb4f469ca598fc58504e5a"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-vq2q"
server_key: "b8596d1150b21cc54379519187315e0a"
server_key_iv: "7b3481cb06036a8b706fe908409813f4"
server_key_version: "encryptionkey@0"
I20260812 02:34:39.896382 1773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "883fb74dd4bb4f469ca598fc58504e5a"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-vq2q"
server_key: "b8596d1150b21cc54379519187315e0a"
server_key_iv: "7b3481cb06036a8b706fe908409813f4"
server_key_version: "encryptionkey@0"
I20260812 02:34:39.902127 1773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260812 02:34:39.906167 1790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:39.907846 1773 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:34:39.908027 1773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "883fb74dd4bb4f469ca598fc58504e5a"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-vq2q"
server_key: "b8596d1150b21cc54379519187315e0a"
server_key_iv: "7b3481cb06036a8b706fe908409813f4"
server_key_version: "encryptionkey@0"
I20260812 02:34:39.908242 1773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:39.939419 1773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:39.940292 1773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:39.940589 1773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:39.941828 1773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:39.943459 1773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:39.943573 1773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:39.943684 1773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:39.943758 1773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:39.980646 1773 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:42769
I20260812 02:34:39.980775 1929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:42769 every 8 connection(s)
I20260812 02:34:39.982065 1773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:39.988945 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 1773
I20260812 02:34:39.989156 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:34:39.989727 12323 external_mini_cluster.cc:1428] Setting key 9273473b7a9836ef69537bbbad1b7420
I20260812 02:34:39.993268 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:37625
--builtin_ntp_servers=127.12.8.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:34:39.996584 1934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:39.985202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:36515 (local address 127.12.8.254:37625)
0812 02:34:39.985801 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:39.985807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:39.987298 (+ 1491us) server_negotiation.cc:407] Connection header received
0812 02:34:39.989282 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:39.989288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:39.989371 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:39.989467 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:39.990793 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.991949 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.993723 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.994178 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.994311 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:39.995705 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:39.995710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:39.995713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:39.995732 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:39.995776 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:39.995789 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:39.995865 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:39.996089 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 02:34:39.996193 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260812 02:34:39.997659 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:39.985914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:37625 (local address 127.12.8.194:36515)
0812 02:34:39.987100 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:39.987151 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:39.988948 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:39.989719 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:39.989728 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:39.989833 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:39.990495 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:39.990520 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.992158 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.992162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:39.993555 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:39.993568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.994414 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.994421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:39.994659 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:39.995381 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:39.995384 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:39.995426 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:39.995503 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:39.995881 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:39.995893 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:39.995901 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:34:39.996056 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 02:34:39.996454 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":369,"threads_started":1}
I20260812 02:34:39.999238 1931 heartbeater.cc:347] Connected to a master server at 127.12.8.254:37625
I20260812 02:34:39.999548 1931 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:40.000312 1931 heartbeater.cc:511] Master 127.12.8.254:37625 requested a full tablet report, sending...
I20260812 02:34:40.001972 1534 ts_manager.cc:194] Registered new tserver with Master: 883fb74dd4bb4f469ca598fc58504e5a (127.12.8.194:42769)
I20260812 02:34:40.003140 1534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.194:36515
W20260812 02:34:40.405124 1936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:40.405512 1936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:40.405647 1936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:40.413240 1936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:40.413367 1936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:40.413511 1936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:34:40.423444 1936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:37625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:40.424938 1936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:40.426946 1936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:40.441987 1941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:40.441968 1945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:40.442951 1942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:40.443445 1936 server_base.cc:1034] running on GCE node
I20260812 02:34:40.444145 1936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:40.445223 1936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:40.446470 1936 hybrid_clock.cc:648] HybridClock initialized: now 1786502080446396 us; error 62 us; skew 500 ppm
I20260812 02:34:40.446820 1936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:40.449337 1936 webserver.cc:460] Webserver started at http://127.12.8.195:37195/ using document root <none> and password file <none>
I20260812 02:34:40.450073 1936 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:40.450181 1936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:40.450515 1936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:40.453648 1936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "5610a23001754d03b55edbb31b3310cf"
format_stamp: "Formatted at 2026-08-12 02:34:40 on dist-test-slave-vq2q"
server_key: "bcc27bcbd6db664a9b6f15ec2b988790"
server_key_iv: "ddcd38f739e4778abb16dfa48d0baf85"
server_key_version: "encryptionkey@0"
I20260812 02:34:40.454468 1936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "5610a23001754d03b55edbb31b3310cf"
format_stamp: "Formatted at 2026-08-12 02:34:40 on dist-test-slave-vq2q"
server_key: "bcc27bcbd6db664a9b6f15ec2b988790"
server_key_iv: "ddcd38f739e4778abb16dfa48d0baf85"
server_key_version: "encryptionkey@0"
I20260812 02:34:40.460227 1936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 02:34:40.464169 1953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:40.466010 1936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:40.466203 1936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "5610a23001754d03b55edbb31b3310cf"
format_stamp: "Formatted at 2026-08-12 02:34:40 on dist-test-slave-vq2q"
server_key: "bcc27bcbd6db664a9b6f15ec2b988790"
server_key_iv: "ddcd38f739e4778abb16dfa48d0baf85"
server_key_version: "encryptionkey@0"
I20260812 02:34:40.466431 1936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:40.467989 1769 heartbeater.cc:503] Master 127.12.8.254:37625 was elected leader, sending a full tablet report...
I20260812 02:34:40.478143 1936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:40.478976 1936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:40.479282 1936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:40.480322 1936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:40.481853 1936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:40.481947 1936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:40.482059 1936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:40.482134 1936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:40.517778 1936 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:43169
I20260812 02:34:40.517897 2084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:43169 every 8 connection(s)
I20260812 02:34:40.519186 1936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:40.527882 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 1936
I20260812 02:34:40.528141 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:34:40.528844 12323 external_mini_cluster.cc:1428] Setting key 96e851e1fcf14c60b1453fc601b2adba
I20260812 02:34:40.535722 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.522366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:52393 (local address 127.12.8.254:37625)
0812 02:34:40.522909 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.522914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.524561 (+ 1647us) server_negotiation.cc:407] Connection header received
0812 02:34:40.526830 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.526836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.526940 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.527062 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:40.528899 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.530082 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.532411 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.532875 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.533000 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.534677 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:40.534683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:40.534688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:40.534721 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:40.534776 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:40.534790 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:40.534871 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.535141 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.535267 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":237,"threads_started":1}
I20260812 02:34:40.537031 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.523087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:37625 (local address 127.12.8.195:52393)
0812 02:34:40.524357 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:40.524405 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:40.526492 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:40.527390 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:40.527403 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:40.527572 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:40.528604 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.528634 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.530273 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.530279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.532231 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.532250 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.533138 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.533146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.533449 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.534283 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:40.534286 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:40.534340 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:40.534451 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:40.534887 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:40.534899 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:40.534908 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:34:40.535100 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 02:34:40.535632 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":430,"threads_started":1}
I20260812 02:34:40.538769 2085 heartbeater.cc:347] Connected to a master server at 127.12.8.254:37625
I20260812 02:34:40.539162 2085 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:40.540124 2085 heartbeater.cc:511] Master 127.12.8.254:37625 requested a full tablet report, sending...
I20260812 02:34:40.541997 1534 ts_manager.cc:194] Registered new tserver with Master: 5610a23001754d03b55edbb31b3310cf (127.12.8.195:43169)
I20260812 02:34:40.542979 1534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.195:52393
I20260812 02:34:40.545445 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:40.556763 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.550242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.12.8.254:37625)
0812 02:34:40.550458 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.550464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.551012 (+ 548us) server_negotiation.cc:407] Connection header received
0812 02:34:40.551180 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.551184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.551256 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.551362 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:40.551891 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.552595 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.554205 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.554612 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.554712 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.555999 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:40.556006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:40.556010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:40.556035 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:40.556106 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:40.556127 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:40.556244 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.556384 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.556471 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 02:34:40.561300 1534 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:40.568240 1534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:40.608932 2099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.582925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45524)
0812 02:34:40.584560 (+ 1635us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:40.584605 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:40.584796 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:40.601578 (+ 16782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:40.601583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:40.601610 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:40.602040 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.602052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.603814 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.603821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.606550 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.606564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.607925 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.607931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.608142 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.608194 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 02:34:40.608336 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 02:34:40.608556 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"mutex_wait_us":3,"thread_start_us":16306,"threads_started":1}
I20260812 02:34:40.609637 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.583616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.12.8.193:34773)
0812 02:34:40.600884 (+ 17268us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.600895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.600929 (+ 34us) server_negotiation.cc:407] Connection header received
0812 02:34:40.601035 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.601044 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.601337 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.601574 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:40.602236 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.603664 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.606743 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.607726 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.607852 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.608169 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.609026 (+ 857us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.609252 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17032,"thread_start_us":16862,"threads_started":1}
I20260812 02:34:40.609997 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.585139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:43169 (local address 127.0.0.1:47762)
0812 02:34:40.598842 (+ 13703us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:40.598870 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:40.599604 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:40.600876 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:40.600880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:40.600914 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:40.601460 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.601474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.603192 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.603197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.606230 (+ 3033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.606250 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.609159 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.609165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.609382 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.609416 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 02:34:40.609504 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:34:40.609645 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13605,"mutex_wait_us":109,"thread_start_us":13224,"threads_started":1}
I20260812 02:34:40.610226 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.583222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42769 (local address 127.0.0.1:44314)
0812 02:34:40.598834 (+ 15612us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:40.598864 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:40.599109 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:40.600690 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:40.600698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:40.600749 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:40.601465 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.601483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.603413 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.603418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.606230 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.606250 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.608423 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.608429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.608624 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.608656 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:34:40.608757 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 02:34:40.608899 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15411,"mutex_wait_us":180,"spinlock_wait_cycles":15104,"thread_start_us":949,"threads_started":1}
I20260812 02:34:40.610728 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.583958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.12.8.194:42769)
0812 02:34:40.599265 (+ 15307us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.599276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.599321 (+ 45us) server_negotiation.cc:407] Connection header received
0812 02:34:40.599460 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.599469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.599828 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.600101 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:40.601721 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.603247 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.606595 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.608190 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.608351 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.608638 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.608956 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.609645 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15118,"thread_start_us":14951,"threads_started":1}
I20260812 02:34:40.611027 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.585525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.12.8.195:43169)
0812 02:34:40.592048 (+ 6523us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.592063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.599029 (+ 6966us) server_negotiation.cc:407] Connection header received
0812 02:34:40.599873 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.599884 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.600263 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.600518 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:40.601721 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.603041 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.606555 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.608190 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.608351 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.608667 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.610390 (+ 1723us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.610615 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6227,"thread_start_us":5962,"threads_started":1}
I20260812 02:34:40.614588 2008 tablet_service.cc:1467] Processing CreateTablet for tablet d8dfae8a86c84ec98e06d58a69e27134 (DEFAULT_TABLE table=test-table [id=2f2fa66bf1ef4b7684114efb2deb3224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:40.614892 1849 tablet_service.cc:1467] Processing CreateTablet for tablet d8dfae8a86c84ec98e06d58a69e27134 (DEFAULT_TABLE table=test-table [id=2f2fa66bf1ef4b7684114efb2deb3224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:40.614892 1682 tablet_service.cc:1467] Processing CreateTablet for tablet d8dfae8a86c84ec98e06d58a69e27134 (DEFAULT_TABLE table=test-table [id=2f2fa66bf1ef4b7684114efb2deb3224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:40.616657 2008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dfae8a86c84ec98e06d58a69e27134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:40.616671 1682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dfae8a86c84ec98e06d58a69e27134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:40.616671 1849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8dfae8a86c84ec98e06d58a69e27134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:40.628485 2105 tablet_bootstrap.cc:492] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Bootstrap starting.
I20260812 02:34:40.631945 2105 tablet_bootstrap.cc:654] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:40.633024 2105 log.cc:824] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:40.634318 2106 tablet_bootstrap.cc:492] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Bootstrap starting.
I20260812 02:34:40.635421 2107 tablet_bootstrap.cc:492] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2: Bootstrap starting.
I20260812 02:34:40.639533 2105 tablet_bootstrap.cc:492] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: No bootstrap required, opened a new log
I20260812 02:34:40.639536 2107 tablet_bootstrap.cc:654] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:40.639882 2105 ts_tablet_manager.cc:1397] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260812 02:34:40.640399 2106 tablet_bootstrap.cc:654] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:40.640861 2107 log.cc:824] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:40.641669 2106 log.cc:824] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:40.648288 2106 tablet_bootstrap.cc:492] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: No bootstrap required, opened a new log
I20260812 02:34:40.648288 2107 tablet_bootstrap.cc:492] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2: No bootstrap required, opened a new log
I20260812 02:34:40.648617 2107 ts_tablet_manager.cc:1397] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260812 02:34:40.648928 2106 ts_tablet_manager.cc:1397] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260812 02:34:40.648931 2105 raft_consensus.cc:348] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.649549 2105 raft_consensus.cc:374] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:40.649677 2105 raft_consensus.cc:729] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5610a23001754d03b55edbb31b3310cf, State: Initialized, Role: FOLLOWER
I20260812 02:34:40.650461 2105 consensus_queue.cc:260] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.654129 2085 heartbeater.cc:503] Master 127.12.8.254:37625 was elected leader, sending a full tablet report...
I20260812 02:34:40.655021 2105 ts_tablet_manager.cc:1428] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260812 02:34:40.659545 2107 raft_consensus.cc:348] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.660109 2107 raft_consensus.cc:374] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:40.660236 2107 raft_consensus.cc:729] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6bddeec97f14e16898795b7342a56c2, State: Initialized, Role: FOLLOWER
I20260812 02:34:40.660445 2106 raft_consensus.cc:348] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.660883 2106 raft_consensus.cc:374] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:40.660990 2106 raft_consensus.cc:729] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 883fb74dd4bb4f469ca598fc58504e5a, State: Initialized, Role: FOLLOWER
I20260812 02:34:40.660985 2107 consensus_queue.cc:260] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.661598 2106 consensus_queue.cc:260] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.663007 2107 ts_tablet_manager.cc:1428] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260812 02:34:40.663962 1931 heartbeater.cc:503] Master 127.12.8.254:37625 was elected leader, sending a full tablet report...
I20260812 02:34:40.665311 2106 ts_tablet_manager.cc:1428] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260812 02:34:40.691959 1770 tablet.cc:2369] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:40.722412 2113 raft_consensus.cc:482] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:40.722813 2113 raft_consensus.cc:504] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.725036 2113 leader_election.cc:288] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6bddeec97f14e16898795b7342a56c2 (127.12.8.193:34773), 5610a23001754d03b55edbb31b3310cf (127.12.8.195:43169)
W20260812 02:34:40.734346 1932 tablet.cc:2369] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:40.735368 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.725756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:33117 (local address 127.12.8.195:43169)
0812 02:34:40.726009 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.726016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.726710 (+ 694us) server_negotiation.cc:407] Connection header received
0812 02:34:40.726974 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.726980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.727076 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.727190 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:40.728147 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.729320 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.732681 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.734004 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.734192 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.734271 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.734948 (+ 677us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.735094 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:34:40.735402 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.725784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:36945 (local address 127.12.8.193:34773)
0812 02:34:40.726065 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.726073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.726828 (+ 755us) server_negotiation.cc:407] Connection header received
0812 02:34:40.726972 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.726978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.727076 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.727190 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:40.728053 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.729263 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.732681 (+ 3418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.734000 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.734192 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.734275 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.735063 (+ 788us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.735193 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:34:40.735776 2115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.725472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:43169 (local address 127.12.8.194:33117)
0812 02:34:40.726510 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:40.726541 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:40.726796 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:40.727227 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:40.727233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:40.727341 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:40.727941 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.727952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.729500 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.729506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.732369 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.732394 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.734274 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.734282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.734629 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.734678 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 02:34:40.734892 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 02:34:40.735063 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":259,"spinlock_wait_cycles":32000,"thread_start_us":721,"threads_started":1}
I20260812 02:34:40.736263 2114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.725572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.12.8.194:36945)
0812 02:34:40.726702 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:40.726725 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:40.726836 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:40.727227 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:40.727233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:40.727264 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:40.727822 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.727838 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.729461 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.729468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.732404 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.732415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.734487 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.734494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.734851 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.734891 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 02:34:40.735614 (+ 723us) client_negotiation.cc:240] Negotiation successful
0812 02:34:40.735906 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":135,"thread_start_us":458,"threads_started":1}
I20260812 02:34:40.737125 2030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dfae8a86c84ec98e06d58a69e27134" candidate_uuid: "883fb74dd4bb4f469ca598fc58504e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5610a23001754d03b55edbb31b3310cf" is_pre_election: true
I20260812 02:34:40.737583 1705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dfae8a86c84ec98e06d58a69e27134" candidate_uuid: "883fb74dd4bb4f469ca598fc58504e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6bddeec97f14e16898795b7342a56c2" is_pre_election: true
I20260812 02:34:40.737830 2030 raft_consensus.cc:2454] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 883fb74dd4bb4f469ca598fc58504e5a in term 0.
I20260812 02:34:40.738123 1705 raft_consensus.cc:2454] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 883fb74dd4bb4f469ca598fc58504e5a in term 0.
I20260812 02:34:40.738715 1793 leader_election.cc:302] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5610a23001754d03b55edbb31b3310cf, 883fb74dd4bb4f469ca598fc58504e5a; no voters:
I20260812 02:34:40.739225 2113 raft_consensus.cc:2781] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:40.739390 2113 raft_consensus.cc:482] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:40.739497 2113 raft_consensus.cc:3037] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:40.742316 2113 raft_consensus.cc:504] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.743072 2113 leader_election.cc:288] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [CANDIDATE]: Term 1 election: Requested vote from peers d6bddeec97f14e16898795b7342a56c2 (127.12.8.193:34773), 5610a23001754d03b55edbb31b3310cf (127.12.8.195:43169)
I20260812 02:34:40.743621 1705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dfae8a86c84ec98e06d58a69e27134" candidate_uuid: "883fb74dd4bb4f469ca598fc58504e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6bddeec97f14e16898795b7342a56c2"
I20260812 02:34:40.743826 1705 raft_consensus.cc:3037] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:40.743805 2030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8dfae8a86c84ec98e06d58a69e27134" candidate_uuid: "883fb74dd4bb4f469ca598fc58504e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5610a23001754d03b55edbb31b3310cf"
I20260812 02:34:40.744150 2030 raft_consensus.cc:3037] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:40.746321 1705 raft_consensus.cc:2454] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 883fb74dd4bb4f469ca598fc58504e5a in term 1.
I20260812 02:34:40.746860 1791 leader_election.cc:302] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 883fb74dd4bb4f469ca598fc58504e5a, d6bddeec97f14e16898795b7342a56c2; no voters:
I20260812 02:34:40.746992 2030 raft_consensus.cc:2454] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 883fb74dd4bb4f469ca598fc58504e5a in term 1.
I20260812 02:34:40.747149 2113 raft_consensus.cc:2781] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:40.747720 2113 raft_consensus.cc:686] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 1 LEADER]: Becoming Leader. State: Replica: 883fb74dd4bb4f469ca598fc58504e5a, State: Running, Role: LEADER
I20260812 02:34:40.748350 2113 consensus_queue.cc:237] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } }
I20260812 02:34:40.754238 1532 catalog_manager.cc:5495] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 883fb74dd4bb4f469ca598fc58504e5a (127.12.8.194). New cstate: current_term: 1 leader_uuid: "883fb74dd4bb4f469ca598fc58504e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "883fb74dd4bb4f469ca598fc58504e5a" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 } health_report { overall_health: UNKNOWN } } }
W20260812 02:34:40.771097 2086 tablet.cc:2369] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:40.793807 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.787654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.12.8.194:42769)
0812 02:34:40.787950 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.787957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.787989 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:34:40.788148 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.788153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.788274 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.788433 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:40.788865 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.789700 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.791213 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.791659 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.791778 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.792378 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:40.793113 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:40.793284 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.793390 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.793485 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 02:34:40.804340 1705 raft_consensus.cc:1261] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 1 FOLLOWER]: Refusing update from remote peer 883fb74dd4bb4f469ca598fc58504e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:40.804534 2030 raft_consensus.cc:1261] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 1 FOLLOWER]: Refusing update from remote peer 883fb74dd4bb4f469ca598fc58504e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:40.805539 2113 consensus_queue.cc:1035] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6bddeec97f14e16898795b7342a56c2" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 34773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:40.805985 2116 consensus_queue.cc:1035] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5610a23001754d03b55edbb31b3310cf" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:40.846529 1533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:40.846899 1533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:40.851709 1533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3b086a319684a3785391690d813c186: Sending DeleteTablet for 3 replicas of tablet d8dfae8a86c84ec98e06d58a69e27134
I20260812 02:34:40.853359 1682 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dfae8a86c84ec98e06d58a69e27134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:40 UTC) from {username='slave'} at 127.0.0.1:45524
I20260812 02:34:40.853677 2008 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dfae8a86c84ec98e06d58a69e27134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:40 UTC) from {username='slave'} at 127.0.0.1:47762
I20260812 02:34:40.854375 1849 tablet_service.cc:1514] Processing DeleteTablet for tablet d8dfae8a86c84ec98e06d58a69e27134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:40 UTC) from {username='slave'} at 127.0.0.1:44314
I20260812 02:34:40.854621 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 1605
I20260812 02:34:40.873588 2137 tablet_replica.cc:330] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: stopping tablet replica
I20260812 02:34:40.874398 2137 raft_consensus.cc:2229] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:40.875144 2137 raft_consensus.cc:2258] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:40.876197 2138 tablet_replica.cc:330] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: stopping tablet replica
I20260812 02:34:40.876782 2138 raft_consensus.cc:2229] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:34:40.877637 2138 raft_consensus.cc:2258] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:40.879172 2137 ts_tablet_manager.cc:1905] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:40.880939 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.856001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.12.8.193:34773)
0812 02:34:40.856238 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.856245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.867383 (+ 11138us) server_negotiation.cc:407] Connection header received
0812 02:34:40.868020 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.868029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.868201 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.868289 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:40.869097 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.870425 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.873481 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.874280 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.874418 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.878978 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:40.878991 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:40.879011 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:40.879880 (+ 869us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:40.879951 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:40.879982 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:40.880085 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.880371 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.880523 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 02:34:40.888628 2136 tablet_replica.cc:330] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2: stopping tablet replica
I20260812 02:34:40.888955 1757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:40.890731 2138 ts_tablet_manager.cc:1905] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:40.899577 2138 ts_tablet_manager.cc:1918] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:40.899842 2138 log.cc:1196] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/d8dfae8a86c84ec98e06d58a69e27134
I20260812 02:34:40.900666 2138 ts_tablet_manager.cc:1939] T d8dfae8a86c84ec98e06d58a69e27134 P 883fb74dd4bb4f469ca598fc58504e5a: Deleting consensus metadata
I20260812 02:34:40.900882 2137 ts_tablet_manager.cc:1918] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:40.901157 2137 log.cc:1196] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/d8dfae8a86c84ec98e06d58a69e27134
I20260812 02:34:40.901839 2137 ts_tablet_manager.cc:1939] T d8dfae8a86c84ec98e06d58a69e27134 P 5610a23001754d03b55edbb31b3310cf: Deleting consensus metadata
I20260812 02:34:40.904017 1519 catalog_manager.cc:4841] TS 883fb74dd4bb4f469ca598fc58504e5a (127.12.8.194:42769): tablet d8dfae8a86c84ec98e06d58a69e27134 (table test-table [id=2f2fa66bf1ef4b7684114efb2deb3224]) successfully deleted
I20260812 02:34:40.904394 1519 catalog_manager.cc:4841] TS 5610a23001754d03b55edbb31b3310cf (127.12.8.195:43169): tablet d8dfae8a86c84ec98e06d58a69e27134 (table test-table [id=2f2fa66bf1ef4b7684114efb2deb3224]) successfully deleted
I20260812 02:34:40.889339 2136 raft_consensus.cc:2229] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:41.020681 2136 raft_consensus.cc:2258] T d8dfae8a86c84ec98e06d58a69e27134 P d6bddeec97f14e16898795b7342a56c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:41.020771 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 1605
I20260812 02:34:41.047636 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 1773
W20260812 02:34:41.047799 1516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.8.193:34773 (error 108)
I20260812 02:34:41.050165 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.049669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45542)
0812 02:34:41.049848 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.049978 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 02:34:41.050614 1516 catalog_manager.cc:4584] TS d6bddeec97f14e16898795b7342a56c2 (127.12.8.193:34773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8dfae8a86c84ec98e06d58a69e27134: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
I20260812 02:34:41.057530 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.048724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.12.8.194:42769)
0812 02:34:41.048967 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:41.048974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:41.049254 (+ 280us) server_negotiation.cc:407] Connection header received
0812 02:34:41.049359 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:41.049366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:41.049500 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:41.049604 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:41.050335 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.051366 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.053505 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.054096 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.054226 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:41.055627 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:41.055637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:41.055657 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:41.056482 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:41.056557 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:41.056581 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:41.056687 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:41.056900 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 02:34:41.057034 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:34:41.058533 1925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:41.104601 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.104001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45556)
0812 02:34:41.104273 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.104379 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260812 02:34:41.106259 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.105830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45558)
0812 02:34:41.106007 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.106107 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 02:34:41.167434 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.166671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45566)
0812 02:34:41.167048 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.167201 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260812 02:34:41.168644 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.168297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45580)
0812 02:34:41.168441 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.168522 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 02:34:41.180411 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 1773
I20260812 02:34:41.207779 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 1936
I20260812 02:34:41.215927 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.208742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.12.8.195:43169)
0812 02:34:41.209030 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:41.209036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:41.209062 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:34:41.209354 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:41.209360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:41.209483 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:41.209590 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:41.210068 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.210922 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.212330 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.212766 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.212867 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:41.214231 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:41.214243 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:41.214268 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:41.215024 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:41.215093 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:41.215120 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:41.215228 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:41.215390 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 02:34:41.215516 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 02:34:41.216806 2079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:41.250065 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.249558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45590)
0812 02:34:41.249773 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.249871 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 02:34:41.251362 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.250952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45592)
0812 02:34:41.251152 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.251258 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260812 02:34:41.322887 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 1936
I20260812 02:34:41.347200 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 1498
I20260812 02:34:41.348322 1568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:41.473241 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.472294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:34773 (local address 127.0.0.1:45608)
0812 02:34:41.472608 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.472730 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260812 02:34:41.473789 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 1498
2026-08-12T02:34:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3300 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:34:41 dist-test-slave-vq2q krb5kdc[2147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:34:41 dist-test-slave-vq2q krb5kdc[2147](info): set up 2 sockets
Aug 12 02:34:41 dist-test-slave-vq2q krb5kdc[2147](info): commencing operation
krb5kdc: starting...
W20260812 02:34:42.649956 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:34:42 dist-test-slave-vq2q krb5kdc[2147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:34:43.045289 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:42673
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:42673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:43.460994 2165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:43.461364 2165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:43.461513 2165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:43.468819 2165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:43.468935 2165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:43.468992 2165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:43.469082 2165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:43.469142 2165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:43.479169 2165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:42673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:42673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:43.480624 2165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:43.482100 2165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:43.494990 2173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:43.495205 2171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:43.494946 2170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:43.496466 2165 server_base.cc:1034] running on GCE node
I20260812 02:34:43.497344 2165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:43.498956 2165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:43.500283 2165 hybrid_clock.cc:648] HybridClock initialized: now 1786502083500214 us; error 60 us; skew 500 ppm
I20260812 02:34:43.500744 2165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:43 dist-test-slave-vq2q krb5kdc[2147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502083, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:43.508246 2165 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:34:43.510464 2165 webserver.cc:460] Webserver started at http://127.12.8.254:46641/ using document root <none> and password file <none>
I20260812 02:34:43.511381 2165 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:43.511509 2165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:43.511848 2165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:43.515964 2165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9"
format_stamp: "Formatted at 2026-08-12 02:34:43 on dist-test-slave-vq2q"
server_key: "088b58996f058d01e3fa0c4f8a889d98"
server_key_iv: "5fdfc8d8ba721033123f719cf386a52b"
server_key_version: "encryptionkey@0"
I20260812 02:34:43.516906 2165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9"
format_stamp: "Formatted at 2026-08-12 02:34:43 on dist-test-slave-vq2q"
server_key: "088b58996f058d01e3fa0c4f8a889d98"
server_key_iv: "5fdfc8d8ba721033123f719cf386a52b"
server_key_version: "encryptionkey@0"
I20260812 02:34:43.524081 2165 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 02:34:43.529140 2183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:43.531169 2165 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:34:43.531422 2165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9"
format_stamp: "Formatted at 2026-08-12 02:34:43 on dist-test-slave-vq2q"
server_key: "088b58996f058d01e3fa0c4f8a889d98"
server_key_iv: "5fdfc8d8ba721033123f719cf386a52b"
server_key_version: "encryptionkey@0"
I20260812 02:34:43.531683 2165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:43.545503 2165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:43.550545 2165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:43.550876 2165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:43.573104 2165 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:42673
I20260812 02:34:43.573143 2245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:42673 every 8 connection(s)
I20260812 02:34:43.574944 2165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:43.576059 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 2165
I20260812 02:34:43.576272 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:34:43.576771 12323 external_mini_cluster.cc:1428] Setting key 22a172b3452fa72bc9d02665a0a2b7b2
I20260812 02:34:43.579839 2246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:43.593768 2246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9: Bootstrap starting.
Aug 12 02:34:43 dist-test-slave-vq2q krb5kdc[2147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:43.597066 2246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:43.598269 2246 log.cc:824] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:43.601464 2246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9: No bootstrap required, opened a new log
I20260812 02:34:43.605238 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.579016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.12.8.254:42673)
0812 02:34:43.580015 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:43.580034 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:43.580098 (+ 64us) server_negotiation.cc:407] Connection header received
0812 02:34:43.582150 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:43.582179 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:43.582716 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:43.583171 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:43.585257 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.586909 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.589014 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.590016 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.590243 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.595991 (+ 5748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:43.596002 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:43.596023 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:43.596087 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:43.599912 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:43.600834 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:43.600842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:43.600850 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:43.601069 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:43.601543 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:43.601547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:43.601549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:43.602321 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:43.602684 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:43.603247 (+ 563us) server_negotiation.cc:299] Negotiation successful
0812 02:34:43.603808 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":442,"threads_started":1}
I20260812 02:34:43.610563 2246 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42673 } }
I20260812 02:34:43.610975 2246 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:43.611066 2246 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e1baabbebd4621a0f86a3f2df2a4d9, State: Initialized, Role: FOLLOWER
I20260812 02:34:43.611656 2246 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42673 } }
I20260812 02:34:43.611886 2246 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:43.612015 2246 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:43.612181 2246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:43.614725 2246 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42673 } }
I20260812 02:34:43.615229 2246 leader_election.cc:302] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5e1baabbebd4621a0f86a3f2df2a4d9; no voters:
I20260812 02:34:43.615897 2246 leader_election.cc:288] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:43.616138 2252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:43.616928 2252 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [term 1 LEADER]: Becoming Leader. State: Replica: e5e1baabbebd4621a0f86a3f2df2a4d9, State: Running, Role: LEADER
I20260812 02:34:43.617719 2252 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42673 } }
I20260812 02:34:43.618168 2246 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:43.621815 2257 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e1baabbebd4621a0f86a3f2df2a4d9. Latest consensus state: current_term: 1 leader_uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42673 } } }
I20260812 02:34:43.621827 2254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e1baabbebd4621a0f86a3f2df2a4d9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 42673 } } }
I20260812 02:34:43.622417 2254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:43.622417 2257 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:43.624744 2264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:43.631224 2264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:43.642109 2264 catalog_manager.cc:1347] Generated new cluster ID: d9a3d2fd2e5f4e21bf07a15ca9977939
I20260812 02:34:43.642261 2264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:43.655653 2264 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:43.657481 2264 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:43.664816 2264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9: Generated new TSK 0
I20260812 02:34:43.666251 2264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:34:43.764910 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42673
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:44.203343 2281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:44.203740 2281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:44.203887 2281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:44.212108 2281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:44.212265 2281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:44.212436 2281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:34:44.224160 2281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:44.226143 2281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:44.227906 2281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:44.243870 2287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:44.243790 2288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:44.243760 2290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:44.246981 2281 server_base.cc:1034] running on GCE node
I20260812 02:34:44.247826 2281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:44.249122 2281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:44.250397 2281 hybrid_clock.cc:648] HybridClock initialized: now 1786502084250313 us; error 74 us; skew 500 ppm
I20260812 02:34:44.250834 2281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:44 dist-test-slave-vq2q krb5kdc[2147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502084, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:44.255645 2281 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:34:44.257779 2281 webserver.cc:460] Webserver started at http://127.12.8.193:36125/ using document root <none> and password file <none>
I20260812 02:34:44.258662 2281 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:44.258790 2281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:44.259141 2281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:44.263028 2281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "3de94a7046b1453ab8b8405c7ffd016f"
format_stamp: "Formatted at 2026-08-12 02:34:44 on dist-test-slave-vq2q"
server_key: "c89a77929558ba820e41d7c28cfe2ddc"
server_key_iv: "076fa6ad545812770fd7bc334f42e9a7"
server_key_version: "encryptionkey@0"
I20260812 02:34:44.263924 2281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "3de94a7046b1453ab8b8405c7ffd016f"
format_stamp: "Formatted at 2026-08-12 02:34:44 on dist-test-slave-vq2q"
server_key: "c89a77929558ba820e41d7c28cfe2ddc"
server_key_iv: "076fa6ad545812770fd7bc334f42e9a7"
server_key_version: "encryptionkey@0"
I20260812 02:34:44.270648 2281 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260812 02:34:44.275215 2298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:44.277035 2281 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:34:44.277297 2281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "3de94a7046b1453ab8b8405c7ffd016f"
format_stamp: "Formatted at 2026-08-12 02:34:44 on dist-test-slave-vq2q"
server_key: "c89a77929558ba820e41d7c28cfe2ddc"
server_key_iv: "076fa6ad545812770fd7bc334f42e9a7"
server_key_version: "encryptionkey@0"
I20260812 02:34:44.277554 2281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:44.327160 2281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:44.332728 2281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:44.333098 2281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:44.334411 2281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:44.336160 2281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:44.336249 2281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:44.336380 2281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:44.336453 2281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:44.374187 2281 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:42239
I20260812 02:34:44.374282 2445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:42239 every 8 connection(s)
I20260812 02:34:44.375646 2281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:44.383435 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 2281
I20260812 02:34:44.383634 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:34:44.384160 12323 external_mini_cluster.cc:1428] Setting key e2b05db8bf7290a8246bfde8a6d407f6
Aug 12 02:34:44 dist-test-slave-vq2q krb5kdc[2147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502084, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:44.401063 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:44.378937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:57743 (local address 127.12.8.254:42673)
0812 02:34:44.379535 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:44.379540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:44.380446 (+ 906us) server_negotiation.cc:407] Connection header received
0812 02:34:44.382991 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:44.382998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:44.383099 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:44.383216 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:44.385525 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:44.386480 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:44.388324 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:44.388921 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:44.389080 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:44.394008 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:44.394014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:44.394018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:44.394061 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:44.396817 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:44.398139 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:44.398144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:44.398146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:44.398303 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:44.398945 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:44.398949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:44.398951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:44.399265 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:44.399394 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:44.400575 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0812 02:34:44.400791 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":341,"threads_started":1}
I20260812 02:34:44.402962 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:44.379353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42673 (local address 127.12.8.193:57743)
0812 02:34:44.380229 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:44.380282 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:44.382539 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:44.383563 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:44.383578 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:44.384367 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:44.385317 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:44.385345 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:44.386685 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:44.386699 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:44.388147 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:44.388162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:44.389220 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:44.389228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:44.389554 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:44.390363 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:44.390391 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:44.393639 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:44.397039 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:44.397051 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:44.397076 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:44.397925 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:44.398436 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:44.398441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:44.398444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:44.398786 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:44.399412 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:44.399422 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:44.400043 (+ 621us) client_negotiation.cc:769] Sending connection context
0812 02:34:44.400429 (+ 386us) client_negotiation.cc:240] Negotiation successful
0812 02:34:44.401154 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":336,"threads_started":1}
I20260812 02:34:44.405125 2446 heartbeater.cc:347] Connected to a master server at 127.12.8.254:42673
I20260812 02:34:44.405514 2446 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:44.406451 2446 heartbeater.cc:511] Master 127.12.8.254:42673 requested a full tablet report, sending...
I20260812 02:34:44.409384 2202 ts_manager.cc:194] Registered new tserver with Master: 3de94a7046b1453ab8b8405c7ffd016f (127.12.8.193:42239)
I20260812 02:34:44.411415 2202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:57743
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:34:44.470734 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42673
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:44.850486 2453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:44.850790 2453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:44.850912 2453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:44.858283 2453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:44.858415 2453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:44.858572 2453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:34:44.868664 2453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:44.870074 2453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:44.871675 2453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:44.886533 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:44.886652 2463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:44.886611 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:44.887908 2453 server_base.cc:1034] running on GCE node
I20260812 02:34:44.888593 2453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:44.889719 2453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:44.891002 2453 hybrid_clock.cc:648] HybridClock initialized: now 1786502084890938 us; error 45 us; skew 500 ppm
I20260812 02:34:44.891368 2453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:44 dist-test-slave-vq2q krb5kdc[2147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502084, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:44.894750 2453 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:34:44.896450 2453 webserver.cc:460] Webserver started at http://127.12.8.194:39889/ using document root <none> and password file <none>
I20260812 02:34:44.897158 2453 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:44.897328 2453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:44.897648 2453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:44.900640 2453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f"
format_stamp: "Formatted at 2026-08-12 02:34:44 on dist-test-slave-vq2q"
server_key: "39664d0968c506dd762f2b7708c9f197"
server_key_iv: "9d8e7f69213d0c74ada40a489164119d"
server_key_version: "encryptionkey@0"
I20260812 02:34:44.901571 2453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f"
format_stamp: "Formatted at 2026-08-12 02:34:44 on dist-test-slave-vq2q"
server_key: "39664d0968c506dd762f2b7708c9f197"
server_key_iv: "9d8e7f69213d0c74ada40a489164119d"
server_key_version: "encryptionkey@0"
I20260812 02:34:44.907306 2453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:34:44.912742 2477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:44.914816 2453 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 02:34:44.915120 2453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f"
format_stamp: "Formatted at 2026-08-12 02:34:44 on dist-test-slave-vq2q"
server_key: "39664d0968c506dd762f2b7708c9f197"
server_key_iv: "9d8e7f69213d0c74ada40a489164119d"
server_key_version: "encryptionkey@0"
I20260812 02:34:44.915525 2453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:44.951959 2453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:44.956337 2453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:44.956625 2453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:44.957780 2453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:44.959312 2453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:44.959408 2453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:44.959542 2453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:44.959597 2453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:44.996644 2453 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:46857
I20260812 02:34:44.996759 2614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:46857 every 8 connection(s)
I20260812 02:34:44.998024 2453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:45.003688 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 2453
I20260812 02:34:45.003897 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:34:45.004418 12323 external_mini_cluster.cc:1428] Setting key 134c672342ef2cf75c05015d22e3dbbd
Aug 12 02:34:45 dist-test-slave-vq2q krb5kdc[2147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502084, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:45.019840 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.001267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:59585 (local address 127.12.8.254:42673)
0812 02:34:45.001823 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.001828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.003013 (+ 1185us) server_negotiation.cc:407] Connection header received
0812 02:34:45.005139 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.005145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.005265 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.005374 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:45.007062 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.007817 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.009463 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.009893 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.010006 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.013966 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:45.013971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:45.013975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:45.014012 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:45.016609 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:45.017409 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:45.017414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:45.017416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:45.017542 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:45.017985 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:45.017988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:45.017990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:45.018292 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:45.018432 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.019202 (+ 770us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.019440 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":313,"threads_started":1}
I20260812 02:34:45.020991 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.001917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42673 (local address 127.12.8.194:59585)
0812 02:34:45.002815 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:45.002875 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:45.004841 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:45.005669 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:45.005678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:45.006245 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:45.006858 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.006877 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.007984 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.007988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.009319 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.009333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.010139 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.010147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.010379 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.011031 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:45.011059 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:45.013694 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:45.016761 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:45.016770 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:45.016788 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:45.017286 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:45.017653 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:45.017658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:45.017662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:45.017867 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:45.018425 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:45.018433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:45.018822 (+ 389us) client_negotiation.cc:769] Sending connection context
0812 02:34:45.019079 (+ 257us) client_negotiation.cc:240] Negotiation successful
0812 02:34:45.019705 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":359,"threads_started":1}
I20260812 02:34:45.022861 2616 heartbeater.cc:347] Connected to a master server at 127.12.8.254:42673
I20260812 02:34:45.023182 2616 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:45.024022 2616 heartbeater.cc:511] Master 127.12.8.254:42673 requested a full tablet report, sending...
I20260812 02:34:45.025610 2202 ts_manager.cc:194] Registered new tserver with Master: 77bcc3e5787d4eb6a7124f94d3eceb9f (127.12.8.194:46857)
I20260812 02:34:45.026728 2202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:59585
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:34:45.094826 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42673
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:45.415645 2446 heartbeater.cc:503] Master 127.12.8.254:42673 was elected leader, sending a full tablet report...
W20260812 02:34:45.445945 2623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:45.446229 2623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:45.446321 2623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:45.453502 2623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:45.453624 2623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:45.453747 2623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:34:45.463672 2623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:42673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:45.465029 2623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:45.466400 2623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:45.480165 2632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:45.480353 2629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:45.480353 2628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:45.481577 2623 server_base.cc:1034] running on GCE node
I20260812 02:34:45.482321 2623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:45.483361 2623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:45.484607 2623 hybrid_clock.cc:648] HybridClock initialized: now 1786502085484536 us; error 69 us; skew 500 ppm
I20260812 02:34:45.484974 2623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:45 dist-test-slave-vq2q krb5kdc[2147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502085, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:45.488420 2623 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:34:45.490103 2623 webserver.cc:460] Webserver started at http://127.12.8.195:40503/ using document root <none> and password file <none>
I20260812 02:34:45.490792 2623 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:45.490912 2623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:45.491240 2623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:45.494242 2623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6"
format_stamp: "Formatted at 2026-08-12 02:34:45 on dist-test-slave-vq2q"
server_key: "0296cb1eba84dfd27d8b0e491e070e64"
server_key_iv: "0d2923c0a7be8c6b9a1cfe950db9791d"
server_key_version: "encryptionkey@0"
I20260812 02:34:45.494995 2623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6"
format_stamp: "Formatted at 2026-08-12 02:34:45 on dist-test-slave-vq2q"
server_key: "0296cb1eba84dfd27d8b0e491e070e64"
server_key_iv: "0d2923c0a7be8c6b9a1cfe950db9791d"
server_key_version: "encryptionkey@0"
I20260812 02:34:45.500828 2623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:34:45.505399 2642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:45.507269 2623 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:34:45.507494 2623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6"
format_stamp: "Formatted at 2026-08-12 02:34:45 on dist-test-slave-vq2q"
server_key: "0296cb1eba84dfd27d8b0e491e070e64"
server_key_iv: "0d2923c0a7be8c6b9a1cfe950db9791d"
server_key_version: "encryptionkey@0"
I20260812 02:34:45.507759 2623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:45.543380 2623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:45.548554 2623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:45.548882 2623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:45.550197 2623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:45.551980 2623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:45.552299 2623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:45.552490 2623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:45.552559 2623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:45.590579 2623 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:46021
I20260812 02:34:45.590652 2779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:46021 every 8 connection(s)
I20260812 02:34:45.592171 2623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:45.596454 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 2623
I20260812 02:34:45.596711 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:34:45.597404 12323 external_mini_cluster.cc:1428] Setting key 28bce13490aef5f857a12463342d244e
Aug 12 02:34:45 dist-test-slave-vq2q krb5kdc[2147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502085, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:45.615697 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.595063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:55639 (local address 127.12.8.254:42673)
0812 02:34:45.595606 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.595618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.596868 (+ 1250us) server_negotiation.cc:407] Connection header received
0812 02:34:45.599392 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.599397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.599472 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.599583 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:45.601677 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.602493 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.604378 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.604779 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.604888 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.608941 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:45.608947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:45.608951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:45.608993 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:45.611781 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:45.613037 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:45.613043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:45.613046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:45.613237 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:45.613875 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:45.613879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:45.613881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:45.614195 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:45.614337 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.615218 (+ 881us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.615406 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":241,"threads_started":1}
I20260812 02:34:45.616976 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.595566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:42673 (local address 127.12.8.195:55639)
0812 02:34:45.596550 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:45.596619 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:45.599008 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:45.599893 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:45.599904 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:45.600631 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:45.601471 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.601496 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.602673 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.602678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.604214 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.604230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.604996 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.605002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.605311 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.606048 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:45.606079 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:45.608678 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:45.611999 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:45.612012 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:45.612034 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:45.612778 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:45.613372 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:45.613377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:45.613380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:45.613730 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:45.614342 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:45.614349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:45.614806 (+ 457us) client_negotiation.cc:769] Sending connection context
0812 02:34:45.615091 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 02:34:45.615626 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":369,"threads_started":1}
I20260812 02:34:45.618566 2780 heartbeater.cc:347] Connected to a master server at 127.12.8.254:42673
I20260812 02:34:45.618937 2780 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:45.619930 2780 heartbeater.cc:511] Master 127.12.8.254:42673 requested a full tablet report, sending...
I20260812 02:34:45.621797 2202 ts_manager.cc:194] Registered new tserver with Master: bb9a7c55a5cb4314a2ef58acbc448aa6 (127.12.8.195:46021)
I20260812 02:34:45.622645 2202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:55639
I20260812 02:34:45.626235 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:45.644250 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.630623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.12.8.254:42673)
0812 02:34:45.630799 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.630803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.631355 (+ 552us) server_negotiation.cc:407] Connection header received
0812 02:34:45.631540 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.631544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.631609 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.631694 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:45.633154 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.633919 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.635270 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.635674 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.635786 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.638739 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:45.638744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:45.638748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:45.638781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:45.641440 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:45.642242 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:45.642246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:45.642248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:45.642392 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:45.642846 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:45.642850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:45.642852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:45.643156 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:45.643438 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.643781 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.643965 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:34:45.648542 2202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:45.655495 2202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:45.694926 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.670322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.12.8.195:46021)
0812 02:34:45.685064 (+ 14742us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.685072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.685111 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:34:45.685264 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.685273 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.685594 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.685912 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:45.686857 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.688636 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.691658 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.693563 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.693689 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.693961 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.694445 (+ 484us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.694630 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14566,"thread_start_us":14354,"threads_started":1}
I20260812 02:34:45.694952 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.670867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.12.8.194:46857)
0812 02:34:45.682513 (+ 11646us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.682524 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.684219 (+ 1695us) server_negotiation.cc:407] Connection header received
0812 02:34:45.684588 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.684599 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.684946 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.685206 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:45.686814 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.688603 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.691295 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.692989 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.693152 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.693448 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.694341 (+ 893us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.694555 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11411,"thread_start_us":12417,"threads_started":1}
I20260812 02:34:45.695011 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.669467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:42239 (local address 127.0.0.1:40050)
0812 02:34:45.679860 (+ 10393us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:45.682586 (+ 2726us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:45.682723 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:45.685333 (+ 2610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:45.685343 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:45.685405 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:45.686387 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.686409 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.688439 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.688445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.691210 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.691230 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.693678 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.693684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.694054 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.694118 (+ 64us) client_negotiation.cc:769] Sending connection context
0812 02:34:45.694204 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 02:34:45.694509 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10066,"mutex_wait_us":172,"spinlock_wait_cycles":1408,"thread_start_us":14274,"threads_started":1}
I20260812 02:34:45.695847 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.671271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:46857 (local address 127.0.0.1:50480)
0812 02:34:45.684077 (+ 12806us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:45.684098 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:45.684350 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:45.685535 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:45.685540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:45.685569 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:45.686563 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.686578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.688808 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.688814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.691099 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.691116 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.693658 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.693667 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.693992 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.694053 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 02:34:45.695240 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0812 02:34:45.695398 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12660,"mutex_wait_us":234,"thread_start_us":8084,"threads_started":1}
I20260812 02:34:45.696159 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.669685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:46021 (local address 127.0.0.1:56624)
0812 02:34:45.680258 (+ 10573us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:45.680311 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:45.680568 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:45.685886 (+ 5318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:45.685891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:45.685915 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:45.686573 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.686583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.688797 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.688803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.691476 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.691487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.693803 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.693810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.694133 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.694163 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:34:45.694261 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 02:34:45.695656 (+ 1395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10461,"mutex_wait_us":1349,"thread_start_us":10628,"threads_started":1}
I20260812 02:34:45.696853 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.670416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.12.8.193:42239)
0812 02:34:45.684589 (+ 14173us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.684599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.684626 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:34:45.684719 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.684725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.684944 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.685211 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:45.686701 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.688287 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.691437 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.693000 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.693151 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.693483 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.696057 (+ 2574us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.696478 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13938,"thread_start_us":13752,"threads_started":1}
I20260812 02:34:45.700542 2699 tablet_service.cc:1467] Processing CreateTablet for tablet bbcec452882240a58744941f9fde5291 (DEFAULT_TABLE table=test-table [id=b10f60f1fc0849a4848562ce47924306]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:45.700582 2534 tablet_service.cc:1467] Processing CreateTablet for tablet bbcec452882240a58744941f9fde5291 (DEFAULT_TABLE table=test-table [id=b10f60f1fc0849a4848562ce47924306]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:45.700946 2363 tablet_service.cc:1467] Processing CreateTablet for tablet bbcec452882240a58744941f9fde5291 (DEFAULT_TABLE table=test-table [id=b10f60f1fc0849a4848562ce47924306]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:45.702306 2699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcec452882240a58744941f9fde5291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:45.702410 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcec452882240a58744941f9fde5291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:45.702412 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcec452882240a58744941f9fde5291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:45.716639 2806 tablet_bootstrap.cc:492] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Bootstrap starting.
I20260812 02:34:45.720516 2809 tablet_bootstrap.cc:492] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Bootstrap starting.
I20260812 02:34:45.720952 2806 tablet_bootstrap.cc:654] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:45.722626 2806 log.cc:824] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:45.725261 2808 tablet_bootstrap.cc:492] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Bootstrap starting.
I20260812 02:34:45.726749 2809 tablet_bootstrap.cc:654] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:45.728358 2809 log.cc:824] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:45.729475 2808 tablet_bootstrap.cc:654] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:45.730824 2808 log.cc:824] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:45.731262 2806 tablet_bootstrap.cc:492] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: No bootstrap required, opened a new log
I20260812 02:34:45.731617 2806 ts_tablet_manager.cc:1397] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20260812 02:34:45.732118 2809 tablet_bootstrap.cc:492] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: No bootstrap required, opened a new log
I20260812 02:34:45.732419 2809 ts_tablet_manager.cc:1397] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260812 02:34:45.734467 2808 tablet_bootstrap.cc:492] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: No bootstrap required, opened a new log
I20260812 02:34:45.734827 2808 ts_tablet_manager.cc:1397] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260812 02:34:45.740538 2806 raft_consensus.cc:348] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
I20260812 02:34:45.740957 2806 raft_consensus.cc:374] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:45.741072 2806 raft_consensus.cc:729] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9a7c55a5cb4314a2ef58acbc448aa6, State: Initialized, Role: FOLLOWER
I20260812 02:34:45.741712 2806 consensus_queue.cc:260] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
I20260812 02:34:45.742990 2780 heartbeater.cc:503] Master 127.12.8.254:42673 was elected leader, sending a full tablet report...
I20260812 02:34:45.743436 2806 ts_tablet_manager.cc:1428] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260812 02:34:45.743906 2809 raft_consensus.cc:348] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
I20260812 02:34:45.744421 2809 raft_consensus.cc:374] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:45.744534 2809 raft_consensus.cc:729] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bcc3e5787d4eb6a7124f94d3eceb9f, State: Initialized, Role: FOLLOWER
I20260812 02:34:45.745244 2809 consensus_queue.cc:260] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
I20260812 02:34:45.746452 2616 heartbeater.cc:503] Master 127.12.8.254:42673 was elected leader, sending a full tablet report...
I20260812 02:34:45.746559 2808 raft_consensus.cc:348] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
I20260812 02:34:45.747219 2809 ts_tablet_manager.cc:1428] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260812 02:34:45.747584 2808 raft_consensus.cc:374] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:45.747699 2808 raft_consensus.cc:729] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de94a7046b1453ab8b8405c7ffd016f, State: Initialized, Role: FOLLOWER
I20260812 02:34:45.748459 2808 consensus_queue.cc:260] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
W20260812 02:34:45.750236 2617 tablet.cc:2369] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:45.751754 2808 ts_tablet_manager.cc:1428] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
I20260812 02:34:45.825111 2816 raft_consensus.cc:482] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:45.825575 2816 raft_consensus.cc:504] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
I20260812 02:34:45.827837 2816 leader_election.cc:288] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb9a7c55a5cb4314a2ef58acbc448aa6 (127.12.8.195:46021), 77bcc3e5787d4eb6a7124f94d3eceb9f (127.12.8.194:46857)
I20260812 02:34:45.838397 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.828457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:54017 (local address 127.12.8.195:46021)
0812 02:34:45.828868 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.828876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.829406 (+ 530us) server_negotiation.cc:407] Connection header received
0812 02:34:45.829678 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.829684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.829780 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.829914 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:45.830785 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.831931 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.835628 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.836882 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.837103 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.837215 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.837968 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.838098 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260812 02:34:45.838713 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.828136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:46021 (local address 127.12.8.193:54017)
0812 02:34:45.829243 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:45.829268 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:45.829497 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:45.829942 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:45.829946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:45.829973 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:45.830547 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.830565 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.832170 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.832178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.835332 (+ 3154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.835362 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.837395 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.837405 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.837709 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.837784 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 02:34:45.837912 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 02:34:45.838123 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":221,"thread_start_us":634,"threads_started":1}
I20260812 02:34:45.838397 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.828536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:40981 (local address 127.12.8.194:46857)
0812 02:34:45.829052 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.829058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.829669 (+ 611us) server_negotiation.cc:407] Connection header received
0812 02:34:45.829816 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.829821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.829900 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.830001 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:45.830799 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.831956 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.835628 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.836957 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.837103 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.837215 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.837955 (+ 740us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.838099 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260812 02:34:45.838719 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.828643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:46857 (local address 127.12.8.193:40981)
0812 02:34:45.829549 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:45.829571 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:45.829690 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:45.830026 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:45.830030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:45.830056 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:45.830610 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.830620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.832174 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.832180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.835332 (+ 3152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:45.835362 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.837412 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.837420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:45.837712 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.837784 (+ 72us) client_negotiation.cc:769] Sending connection context
0812 02:34:45.837912 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 02:34:45.838123 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":272,"thread_start_us":569,"threads_started":1}
I20260812 02:34:45.840324 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcec452882240a58744941f9fde5291" candidate_uuid: "3de94a7046b1453ab8b8405c7ffd016f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" is_pre_election: true
I20260812 02:34:45.840250 2723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcec452882240a58744941f9fde5291" candidate_uuid: "3de94a7046b1453ab8b8405c7ffd016f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" is_pre_election: true
I20260812 02:34:45.840924 2559 raft_consensus.cc:2454] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de94a7046b1453ab8b8405c7ffd016f in term 0.
I20260812 02:34:45.840932 2723 raft_consensus.cc:2454] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3de94a7046b1453ab8b8405c7ffd016f in term 0.
I20260812 02:34:45.841790 2304 leader_election.cc:302] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3de94a7046b1453ab8b8405c7ffd016f, 77bcc3e5787d4eb6a7124f94d3eceb9f, bb9a7c55a5cb4314a2ef58acbc448aa6; no voters:
I20260812 02:34:45.842267 2816 raft_consensus.cc:2781] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:45.842433 2816 raft_consensus.cc:482] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:45.842550 2816 raft_consensus.cc:3037] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 0 FOLLOWER]: Advancing to term 1
W20260812 02:34:45.844070 2781 tablet.cc:2369] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:45.844879 2816 raft_consensus.cc:504] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
I20260812 02:34:45.845551 2816 leader_election.cc:288] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [CANDIDATE]: Term 1 election: Requested vote from peers bb9a7c55a5cb4314a2ef58acbc448aa6 (127.12.8.195:46021), 77bcc3e5787d4eb6a7124f94d3eceb9f (127.12.8.194:46857)
I20260812 02:34:45.846271 2723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcec452882240a58744941f9fde5291" candidate_uuid: "3de94a7046b1453ab8b8405c7ffd016f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6"
I20260812 02:34:45.846272 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcec452882240a58744941f9fde5291" candidate_uuid: "3de94a7046b1453ab8b8405c7ffd016f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f"
I20260812 02:34:45.846557 2723 raft_consensus.cc:3037] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:45.846575 2559 raft_consensus.cc:3037] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:45.850274 2723 raft_consensus.cc:2454] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de94a7046b1453ab8b8405c7ffd016f in term 1.
I20260812 02:34:45.850272 2559 raft_consensus.cc:2454] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3de94a7046b1453ab8b8405c7ffd016f in term 1.
I20260812 02:34:45.850894 2304 leader_election.cc:302] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3de94a7046b1453ab8b8405c7ffd016f, 77bcc3e5787d4eb6a7124f94d3eceb9f, bb9a7c55a5cb4314a2ef58acbc448aa6; no voters:
I20260812 02:34:45.851275 2816 raft_consensus.cc:2781] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:45.851802 2816 raft_consensus.cc:686] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 1 LEADER]: Becoming Leader. State: Replica: 3de94a7046b1453ab8b8405c7ffd016f, State: Running, Role: LEADER
I20260812 02:34:45.852360 2816 consensus_queue.cc:237] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } }
I20260812 02:34:45.857565 2201 catalog_manager.cc:5495] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3de94a7046b1453ab8b8405c7ffd016f (127.12.8.193). New cstate: current_term: 1 leader_uuid: "3de94a7046b1453ab8b8405c7ffd016f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3de94a7046b1453ab8b8405c7ffd016f" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 } health_report { overall_health: UNKNOWN } } }
W20260812 02:34:45.878937 2448 tablet.cc:2369] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:45.892849 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.886807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.12.8.193:42239)
0812 02:34:45.887119 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.887125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.887154 (+ 29us) server_negotiation.cc:407] Connection header received
0812 02:34:45.887394 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.887399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.887517 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.887688 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:45.888089 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.888933 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.890444 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.890944 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.891062 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.891551 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:45.892210 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:45.892359 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.892454 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.892515 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 02:34:45.902700 2559 raft_consensus.cc:1261] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 1 FOLLOWER]: Refusing update from remote peer 3de94a7046b1453ab8b8405c7ffd016f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:45.902813 2723 raft_consensus.cc:1261] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 1 FOLLOWER]: Refusing update from remote peer 3de94a7046b1453ab8b8405c7ffd016f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:45.903995 2820 consensus_queue.cc:1035] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9a7c55a5cb4314a2ef58acbc448aa6" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:45.904443 2816 consensus_queue.cc:1035] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bcc3e5787d4eb6a7124f94d3eceb9f" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 46857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:45.918356 2823 mvcc.cc:205] Tried to move back new op lower bound from 7317512543846346752 to 7317512543655288832. Current Snapshot: MvccSnapshot[applied={T|T < 7317512543846346752}]
I20260812 02:34:45.920552 2826 mvcc.cc:205] Tried to move back new op lower bound from 7317512543846346752 to 7317512543655288832. Current Snapshot: MvccSnapshot[applied={T|T < 7317512543846346752}]
I20260812 02:34:45.937808 2202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:45.938261 2202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:45.943209 2202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5e1baabbebd4621a0f86a3f2df2a4d9: Sending DeleteTablet for 3 replicas of tablet bbcec452882240a58744941f9fde5291
I20260812 02:34:45.944823 2699 tablet_service.cc:1514] Processing DeleteTablet for tablet bbcec452882240a58744941f9fde5291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:45 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:56624
I20260812 02:34:45.944809 2363 tablet_service.cc:1514] Processing DeleteTablet for tablet bbcec452882240a58744941f9fde5291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:45 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:40050
I20260812 02:34:45.945013 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet bbcec452882240a58744941f9fde5291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:45 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:50480
I20260812 02:34:45.947609 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 2281
I20260812 02:34:45.956434 2837 tablet_replica.cc:330] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: stopping tablet replica
I20260812 02:34:45.957366 2837 raft_consensus.cc:2229] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:34:45.958222 2834 tablet_replica.cc:330] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: stopping tablet replica
I20260812 02:34:45.959141 2837 raft_consensus.cc:2258] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:45.959793 2834 raft_consensus.cc:2229] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:45.960832 2834 raft_consensus.cc:2258] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:45.963583 2834 ts_tablet_manager.cc:1905] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:45.966462 2837 ts_tablet_manager.cc:1905] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:45.972179 2834 ts_tablet_manager.cc:1918] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:45.972569 2834 log.cc:1196] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/bbcec452882240a58744941f9fde5291
I20260812 02:34:45.973335 2834 ts_tablet_manager.cc:1939] T bbcec452882240a58744941f9fde5291 P bb9a7c55a5cb4314a2ef58acbc448aa6: Deleting consensus metadata
I20260812 02:34:45.975944 2184 catalog_manager.cc:4841] TS bb9a7c55a5cb4314a2ef58acbc448aa6 (127.12.8.195:46021): tablet bbcec452882240a58744941f9fde5291 (table test-table [id=b10f60f1fc0849a4848562ce47924306]) successfully deleted
I20260812 02:34:45.979981 2837 ts_tablet_manager.cc:1918] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:45.980253 2837 log.cc:1196] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/bbcec452882240a58744941f9fde5291
I20260812 02:34:45.981079 2837 ts_tablet_manager.cc:1939] T bbcec452882240a58744941f9fde5291 P 3de94a7046b1453ab8b8405c7ffd016f: Deleting consensus metadata
I20260812 02:34:45.984010 2188 catalog_manager.cc:4841] TS 3de94a7046b1453ab8b8405c7ffd016f (127.12.8.193:42239): tablet bbcec452882240a58744941f9fde5291 (table test-table [id=b10f60f1fc0849a4848562ce47924306]) successfully deleted
I20260812 02:34:45.985103 2836 tablet_replica.cc:330] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: stopping tablet replica
I20260812 02:34:45.985926 2836 raft_consensus.cc:2229] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:45.986630 2836 raft_consensus.cc:2258] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 12 02:34:45 dist-test-slave-vq2q krb5kdc[2147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:34:45.989675 2836 ts_tablet_manager.cc:1905] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:45.994935 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:45.949238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.12.8.193:42239)
0812 02:34:45.949487 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:45.949495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:45.970373 (+ 20878us) server_negotiation.cc:407] Connection header received
0812 02:34:45.970947 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:45.970955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:45.971061 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:45.971139 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:45.973254 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.974715 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.982654 (+ 7939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:45.983445 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:45.983575 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:45.988581 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:45.988596 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:45.988612 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:45.988663 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:45.991836 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:45.992546 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:45.992553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:45.992555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:45.992715 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:45.993087 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:45.993090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:45.993093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:45.993541 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:45.993863 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:45.994146 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 02:34:45.994421 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 02:34:45.995647 2440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:45.998021 2836 ts_tablet_manager.cc:1918] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:45.998230 2836 log.cc:1196] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/bbcec452882240a58744941f9fde5291
I20260812 02:34:45.998899 2836 ts_tablet_manager.cc:1939] T bbcec452882240a58744941f9fde5291 P 77bcc3e5787d4eb6a7124f94d3eceb9f: Deleting consensus metadata
I20260812 02:34:46.001039 2185 catalog_manager.cc:4841] TS 77bcc3e5787d4eb6a7124f94d3eceb9f (127.12.8.194:46857): tablet bbcec452882240a58744941f9fde5291 (table test-table [id=b10f60f1fc0849a4848562ce47924306]) successfully deleted
I20260812 02:34:46.113268 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 2281
I20260812 02:34:46.140291 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 2453
Aug 12 02:34:46 dist-test-slave-vq2q krb5kdc[2147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:34:46.156293 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.141350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.12.8.194:46857)
0812 02:34:46.141580 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:46.141588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:46.141606 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:34:46.141663 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:46.141667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:46.141734 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:46.141824 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:46.143197 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.144009 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.145521 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.145932 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.146047 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.150395 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:46.150413 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:46.150431 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:46.150472 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:46.153409 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.154135 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.154142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.154145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.154285 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.154656 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.154660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.154662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.155074 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:46.155383 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:46.155708 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 02:34:46.155932 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 02:34:46.157310 2608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:46.256735 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 2453
I20260812 02:34:46.284230 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 2623
Aug 12 02:34:46 dist-test-slave-vq2q krb5kdc[2147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:34:46.303040 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.285337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.12.8.195:46021)
0812 02:34:46.285580 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:46.285587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:46.285606 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:34:46.285666 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:46.285669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:46.285768 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:46.285879 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:46.287352 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.288066 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.289417 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.289804 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.289916 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.294848 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:46.294865 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:46.294883 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:46.294918 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:46.299105 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.300201 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.300213 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.300216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.300430 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.301095 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.301100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.301103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.301640 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:46.301991 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:46.302360 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 02:34:46.302636 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 02:34:46.304626 2775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:46.398836 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 2623
I20260812 02:34:46.419930 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 2165
I20260812 02:34:46.421522 2240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:46.538846 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 2165
2026-08-12T02:34:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5061 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T02:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:46Z Disabled control of system clock
I20260812 02:34:46.655781 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:41331
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:41331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:34:47.085974 2846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:47.086378 2846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:47.086510 2846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:47.094791 2846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:47.094952 2846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:47.094992 2846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:47.095059 2846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:47.095110 2846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:47.105822 2846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:41331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:47.107607 2846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:47.109998 2846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:47.125219 2852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:47.125332 2853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:47.125264 2855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:47.131179 2846 server_base.cc:1034] running on GCE node
I20260812 02:34:47.132364 2846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:47.134426 2846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:47.135708 2846 hybrid_clock.cc:648] HybridClock initialized: now 1786502087135620 us; error 67 us; skew 500 ppm
I20260812 02:34:47.136250 2846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:47.139535 2846 webserver.cc:460] Webserver started at http://127.12.8.254:38791/ using document root <none> and password file <none>
I20260812 02:34:47.140507 2846 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:47.140654 2846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:47.141119 2846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:47.145108 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "362dd0148de94ef59b9ae51f2cb955dc"
format_stamp: "Formatted at 2026-08-12 02:34:47 on dist-test-slave-vq2q"
server_key: "5168304363e2ca8108402226b774cbad"
server_key_iv: "7101791dda401db8429925b4b209e5cd"
server_key_version: "encryptionkey@0"
I20260812 02:34:47.146243 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "362dd0148de94ef59b9ae51f2cb955dc"
format_stamp: "Formatted at 2026-08-12 02:34:47 on dist-test-slave-vq2q"
server_key: "5168304363e2ca8108402226b774cbad"
server_key_iv: "7101791dda401db8429925b4b209e5cd"
server_key_version: "encryptionkey@0"
I20260812 02:34:47.153693 2846 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 02:34:47.158484 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:47.160472 2846 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:34:47.160734 2846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "362dd0148de94ef59b9ae51f2cb955dc"
format_stamp: "Formatted at 2026-08-12 02:34:47 on dist-test-slave-vq2q"
server_key: "5168304363e2ca8108402226b774cbad"
server_key_iv: "7101791dda401db8429925b4b209e5cd"
server_key_version: "encryptionkey@0"
I20260812 02:34:47.161021 2846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:47.209836 2846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:47.211074 2846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:47.211443 2846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:47.234961 2846 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:41331
I20260812 02:34:47.235011 2929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:41331 every 8 connection(s)
I20260812 02:34:47.236860 2846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:47.241302 2931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:47.243199 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 2846
I20260812 02:34:47.243420 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:34:47.244021 12323 external_mini_cluster.cc:1428] Setting key 7b421a6949c8e0ab226a080c9d5ee187
I20260812 02:34:47.254276 2931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc: Bootstrap starting.
I20260812 02:34:47.257627 2931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:47.258735 2931 log.cc:824] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:47.261545 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.246647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.12.8.254:41331)
0812 02:34:47.248027 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.248048 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.248108 (+ 60us) server_negotiation.cc:407] Connection header received
0812 02:34:47.250387 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.250421 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.251231 (+ 810us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.251823 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.252564 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.254116 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.255733 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.256630 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.256839 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.258325 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.258334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.258353 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:47.259073 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.259193 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:47.259221 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.259367 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.259773 (+ 406us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.260250 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":456,"threads_started":1}
I20260812 02:34:47.262269 2931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc: No bootstrap required, opened a new log
I20260812 02:34:47.270972 2931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362dd0148de94ef59b9ae51f2cb955dc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 41331 } }
I20260812 02:34:47.271368 2931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:47.271466 2931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362dd0148de94ef59b9ae51f2cb955dc, State: Initialized, Role: FOLLOWER
I20260812 02:34:47.271988 2931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362dd0148de94ef59b9ae51f2cb955dc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 41331 } }
I20260812 02:34:47.272213 2931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:47.272341 2931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:47.272500 2931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:47.274713 2931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362dd0148de94ef59b9ae51f2cb955dc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 41331 } }
I20260812 02:34:47.275229 2931 leader_election.cc:302] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 362dd0148de94ef59b9ae51f2cb955dc; no voters:
I20260812 02:34:47.275856 2931 leader_election.cc:288] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:47.276111 2937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:47.277302 2937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [term 1 LEADER]: Becoming Leader. State: Replica: 362dd0148de94ef59b9ae51f2cb955dc, State: Running, Role: LEADER
I20260812 02:34:47.278013 2937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362dd0148de94ef59b9ae51f2cb955dc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 41331 } }
I20260812 02:34:47.278223 2931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:47.281585 2938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "362dd0148de94ef59b9ae51f2cb955dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362dd0148de94ef59b9ae51f2cb955dc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 41331 } } }
I20260812 02:34:47.282032 2938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:47.282532 2939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362dd0148de94ef59b9ae51f2cb955dc. Latest consensus state: current_term: 1 leader_uuid: "362dd0148de94ef59b9ae51f2cb955dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362dd0148de94ef59b9ae51f2cb955dc" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 41331 } } }
I20260812 02:34:47.282898 2939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:47.283815 2946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:47.290374 2946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:47.301887 2946 catalog_manager.cc:1347] Generated new cluster ID: 4a2e7442f61f4eba888548388c910911
I20260812 02:34:47.302030 2946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:47.336711 2946 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:47.338124 2946 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:47.351150 2946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc: Generated new TSK 0
I20260812 02:34:47.352094 2946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:34:47.367219 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:41331
--builtin_ntp_servers=127.12.8.212:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:34:47.760811 2957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:47.761108 2957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:47.761221 2957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:47.768772 2957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:47.768878 2957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:47.768985 2957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:34:47.780601 2957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:47.782006 2957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:47.783933 2957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:47.797297 2962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:47.797266 2963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:47.797230 2965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:47.797940 2957 server_base.cc:1034] running on GCE node
I20260812 02:34:47.798533 2957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:47.799465 2957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:47.800688 2957 hybrid_clock.cc:648] HybridClock initialized: now 1786502087800635 us; error 37 us; skew 500 ppm
I20260812 02:34:47.801025 2957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:47.803303 2957 webserver.cc:460] Webserver started at http://127.12.8.193:40119/ using document root <none> and password file <none>
I20260812 02:34:47.803994 2957 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:47.804095 2957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:47.804414 2957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:47.807260 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556"
format_stamp: "Formatted at 2026-08-12 02:34:47 on dist-test-slave-vq2q"
server_key: "0115aa75fb2e999bc803536e1715fb53"
server_key_iv: "547d61de20c1e35ed0da24288f3bfe60"
server_key_version: "encryptionkey@0"
I20260812 02:34:47.808043 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556"
format_stamp: "Formatted at 2026-08-12 02:34:47 on dist-test-slave-vq2q"
server_key: "0115aa75fb2e999bc803536e1715fb53"
server_key_iv: "547d61de20c1e35ed0da24288f3bfe60"
server_key_version: "encryptionkey@0"
I20260812 02:34:47.813242 2957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260812 02:34:47.816686 2972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:47.818069 2957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 02:34:47.818220 2957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556"
format_stamp: "Formatted at 2026-08-12 02:34:47 on dist-test-slave-vq2q"
server_key: "0115aa75fb2e999bc803536e1715fb53"
server_key_iv: "547d61de20c1e35ed0da24288f3bfe60"
server_key_version: "encryptionkey@0"
I20260812 02:34:47.818431 2957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:47.838066 2957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:47.838925 2957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:47.839223 2957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:47.840242 2957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:47.841826 2957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:47.841924 2957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:47.842034 2957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:47.842106 2957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:47.872639 2957 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:43045
I20260812 02:34:47.872717 3119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:43045 every 8 connection(s)
I20260812 02:34:47.873999 2957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:47.883996 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 2957
I20260812 02:34:47.884167 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:34:47.884660 12323 external_mini_cluster.cc:1428] Setting key 2b3f805fd104b3b1e22979443d3fd179
I20260812 02:34:47.886137 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.876514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:48197 (local address 127.12.8.254:41331)
0812 02:34:47.876981 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.876988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.878137 (+ 1149us) server_negotiation.cc:407] Connection header received
0812 02:34:47.879877 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.879881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.879951 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.880038 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.881049 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.881822 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.883314 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.883704 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.883816 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.885246 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.885252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.885257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:47.885277 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.885318 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:47.885330 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.885409 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.885658 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.885773 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
I20260812 02:34:47.887060 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.877022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:41331 (local address 127.12.8.193:48197)
0812 02:34:47.877999 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:47.878042 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:47.879620 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:47.880201 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:47.880208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:47.880287 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:47.880892 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.880909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.881961 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.881965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.883173 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.883185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.883940 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.883946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.884194 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.884908 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:47.884911 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:47.884948 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:47.885025 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:47.885421 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:47.885433 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:47.885441 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:34:47.885635 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 02:34:47.886013 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":323,"threads_started":1}
I20260812 02:34:47.887646 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:41331
--builtin_ntp_servers=127.12.8.212:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:34:47.888581 3122 heartbeater.cc:347] Connected to a master server at 127.12.8.254:41331
I20260812 02:34:47.888904 3122 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:47.889850 3122 heartbeater.cc:511] Master 127.12.8.254:41331 requested a full tablet report, sending...
I20260812 02:34:47.891990 2883 ts_manager.cc:194] Registered new tserver with Master: a8c4caaa44aa45b7b0f3b93b2e0ef556 (127.12.8.193:43045)
I20260812 02:34:47.893487 2883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.193:48197
W20260812 02:34:48.238409 3128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:48.238700 3128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:48.238828 3128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:48.246335 3128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:48.246446 3128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:48.246585 3128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:34:48.255935 3128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:48.257323 3128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:48.259087 3128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:48.273237 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:48.273303 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:48.273272 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:48.274155 3128 server_base.cc:1034] running on GCE node
I20260812 02:34:48.274766 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:48.275784 3128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:48.276994 3128 hybrid_clock.cc:648] HybridClock initialized: now 1786502088276950 us; error 32 us; skew 500 ppm
I20260812 02:34:48.277357 3128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:48.279609 3128 webserver.cc:460] Webserver started at http://127.12.8.194:41509/ using document root <none> and password file <none>
I20260812 02:34:48.280294 3128 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:48.280435 3128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:48.280723 3128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:48.283588 3128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "c30b824bc4a340ebb03d623536d0fc80"
format_stamp: "Formatted at 2026-08-12 02:34:48 on dist-test-slave-vq2q"
server_key: "5daceeeecd6b64f99e0e4d568f18b861"
server_key_iv: "6ef11ba15b3eb89d11a309ac4bb42281"
server_key_version: "encryptionkey@0"
I20260812 02:34:48.284358 3128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "c30b824bc4a340ebb03d623536d0fc80"
format_stamp: "Formatted at 2026-08-12 02:34:48 on dist-test-slave-vq2q"
server_key: "5daceeeecd6b64f99e0e4d568f18b861"
server_key_iv: "6ef11ba15b3eb89d11a309ac4bb42281"
server_key_version: "encryptionkey@0"
I20260812 02:34:48.289570 3128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 02:34:48.293081 3142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:48.294664 3128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:34:48.294821 3128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "c30b824bc4a340ebb03d623536d0fc80"
format_stamp: "Formatted at 2026-08-12 02:34:48 on dist-test-slave-vq2q"
server_key: "5daceeeecd6b64f99e0e4d568f18b861"
server_key_iv: "6ef11ba15b3eb89d11a309ac4bb42281"
server_key_version: "encryptionkey@0"
I20260812 02:34:48.295018 3128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:48.336012 3128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:48.336872 3128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:48.337167 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:48.338218 3128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:48.339731 3128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:48.339831 3128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:48.339941 3128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:48.340014 3128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:48.371577 3128 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:35405
I20260812 02:34:48.371618 3280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:35405 every 8 connection(s)
I20260812 02:34:48.372855 3128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:48.382517 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 3128
I20260812 02:34:48.382715 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:34:48.383247 12323 external_mini_cluster.cc:1428] Setting key 7786c4c4e7414ed3b424677ca532924b
I20260812 02:34:48.386458 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.375493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:46879 (local address 127.12.8.254:41331)
0812 02:34:48.375671 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:48.375676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:48.377155 (+ 1479us) server_negotiation.cc:407] Connection header received
0812 02:34:48.379106 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:48.379113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:48.379195 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:48.379309 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:48.380548 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.381385 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.383154 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.383778 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.383909 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.385431 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:48.385436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:48.385439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:48.385456 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:48.385491 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:48.385505 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:48.385601 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:48.385977 (+ 376us) server_negotiation.cc:299] Negotiation successful
0812 02:34:48.386110 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:34:48.386843 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:41331
--builtin_ntp_servers=127.12.8.212:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:34:48.387794 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.375960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:41331 (local address 127.12.8.194:46879)
0812 02:34:48.376971 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:48.377019 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:48.378810 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:48.379568 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:48.379577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:48.379690 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:48.380339 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.380357 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.381571 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.381575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.382998 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.383012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.384065 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.384074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.384348 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.385067 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:48.385070 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:48.385116 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:48.385242 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:48.385612 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:48.385622 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:48.385628 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:34:48.385977 (+ 349us) client_negotiation.cc:240] Negotiation successful
0812 02:34:48.386477 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":316,"threads_started":1}
I20260812 02:34:48.389509 3281 heartbeater.cc:347] Connected to a master server at 127.12.8.254:41331
I20260812 02:34:48.389879 3281 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:48.390867 3281 heartbeater.cc:511] Master 127.12.8.254:41331 requested a full tablet report, sending...
I20260812 02:34:48.392599 2883 ts_manager.cc:194] Registered new tserver with Master: c30b824bc4a340ebb03d623536d0fc80 (127.12.8.194:35405)
I20260812 02:34:48.393640 2883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.194:46879
W20260812 02:34:48.742252 3285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:48.742565 3285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:48.742681 3285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:48.750136 3285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:48.750257 3285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:48.750393 3285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:34:48.759773 3285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:48.761315 3285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:48.763221 3285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:48.777248 3292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:48.777323 3296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:48.777344 3293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:48.784238 3285 server_base.cc:1034] running on GCE node
I20260812 02:34:48.785163 3285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:48.786552 3285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:48.787828 3285 hybrid_clock.cc:648] HybridClock initialized: now 1786502088787766 us; error 52 us; skew 500 ppm
I20260812 02:34:48.788245 3285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:48.795009 3285 webserver.cc:460] Webserver started at http://127.12.8.195:39729/ using document root <none> and password file <none>
I20260812 02:34:48.795943 3285 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:48.796056 3285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:48.796379 3285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:48.800136 3285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "31089272f8524fe2b98d851e5b9ff785"
format_stamp: "Formatted at 2026-08-12 02:34:48 on dist-test-slave-vq2q"
server_key: "7604ee9a17cda263f595e40916dcdf69"
server_key_iv: "76b3af12657df760cf71640acfcba10e"
server_key_version: "encryptionkey@0"
I20260812 02:34:48.801079 3285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "31089272f8524fe2b98d851e5b9ff785"
format_stamp: "Formatted at 2026-08-12 02:34:48 on dist-test-slave-vq2q"
server_key: "7604ee9a17cda263f595e40916dcdf69"
server_key_iv: "76b3af12657df760cf71640acfcba10e"
server_key_version: "encryptionkey@0"
I20260812 02:34:48.807773 3285 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:34:48.812553 3302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:48.814424 3285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:48.814661 3285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "31089272f8524fe2b98d851e5b9ff785"
format_stamp: "Formatted at 2026-08-12 02:34:48 on dist-test-slave-vq2q"
server_key: "7604ee9a17cda263f595e40916dcdf69"
server_key_iv: "76b3af12657df760cf71640acfcba10e"
server_key_version: "encryptionkey@0"
I20260812 02:34:48.814913 3285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:48.849370 3285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:48.850494 3285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:48.850850 3285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:48.852133 3285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:48.854121 3285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:48.854233 3285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:48.854353 3285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:48.854430 3285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:48.891769 3285 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:44829
I20260812 02:34:48.891860 3450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:44829 every 8 connection(s)
I20260812 02:34:48.893422 3285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:48.895234 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 3285
I20260812 02:34:48.895438 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:34:48.896054 12323 external_mini_cluster.cc:1428] Setting key 5c2ec4b03de78849dfbfce233cf6f543
I20260812 02:34:48.896752 3122 heartbeater.cc:503] Master 127.12.8.254:41331 was elected leader, sending a full tablet report...
I20260812 02:34:48.908043 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.896495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:59871 (local address 127.12.8.254:41331)
0812 02:34:48.897318 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:48.897325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:48.899066 (+ 1741us) server_negotiation.cc:407] Connection header received
0812 02:34:48.901253 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:48.901258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:48.901356 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:48.901453 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:48.902589 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.903507 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.905138 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.905619 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.905748 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.907150 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:48.907156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:48.907161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:48.907185 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:48.907248 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:48.907261 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:48.907336 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:48.907557 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 02:34:48.907663 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":456,"threads_started":1}
I20260812 02:34:48.909068 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.897446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:41331 (local address 127.12.8.195:59871)
0812 02:34:48.898845 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:48.898902 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:48.900933 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:48.901704 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:48.901713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:48.901819 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:48.902425 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.902442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.903672 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.903676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.904986 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.905000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.905883 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.905891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.906112 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.906815 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:48.906818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:48.906861 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:48.906945 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:48.907355 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:48.907367 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:48.907373 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:34:48.907534 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 02:34:48.907951 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":443,"threads_started":1}
I20260812 02:34:48.911159 3451 heartbeater.cc:347] Connected to a master server at 127.12.8.254:41331
I20260812 02:34:48.911533 3451 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:48.912488 3451 heartbeater.cc:511] Master 127.12.8.254:41331 requested a full tablet report, sending...
I20260812 02:34:48.914181 2882 ts_manager.cc:194] Registered new tserver with Master: 31089272f8524fe2b98d851e5b9ff785 (127.12.8.195:44829)
I20260812 02:34:48.915287 2882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.195:59871
I20260812 02:34:48.927177 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:48.939685 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.932555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.12.8.254:41331)
0812 02:34:48.932777 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:48.932783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:48.933288 (+ 505us) server_negotiation.cc:407] Connection header received
0812 02:34:48.933422 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:48.933426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:48.933523 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:48.933654 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:48.934256 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.935086 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.936758 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.937271 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.937426 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.938846 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:48.938853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:48.938859 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:48.938886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:48.938940 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:48.938954 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:48.939092 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:48.939331 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 02:34:48.939421 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:34:48.945591 2882 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:48.953924 2882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:48.992313 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.970802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:35405 (local address 127.0.0.1:56444)
0812 02:34:48.972630 (+ 1828us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:48.972683 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:48.972917 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:48.975922 (+ 3005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:48.975933 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:48.975979 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:48.976588 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.976603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.981396 (+ 4793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.981404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.987263 (+ 5859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.987295 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.991102 (+ 3807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.991111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.991464 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.991573 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 02:34:48.991740 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 02:34:48.991958 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1548,"mutex_wait_us":30,"thread_start_us":1334,"threads_started":1}
I20260812 02:34:48.993942 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.971528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.12.8.194:35405)
0812 02:34:48.972319 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:48.972345 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:48.972823 (+ 478us) server_negotiation.cc:407] Connection header received
0812 02:34:48.973280 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:48.973289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:48.973660 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:48.973895 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:48.977106 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.981074 (+ 3968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.987711 (+ 6637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.990407 (+ 2696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.990579 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.990942 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:48.993154 (+ 2212us) server_negotiation.cc:299] Negotiation successful
0812 02:34:48.993482 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":742,"threads_started":1}
I20260812 02:34:48.995204 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.971572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:43045 (local address 127.0.0.1:48074)
0812 02:34:48.984230 (+ 12658us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:48.984257 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:48.984503 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:48.986057 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:48.986063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:48.986102 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:48.986776 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.986795 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.989129 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.989135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.991355 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.991380 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.993626 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.993636 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.993909 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.993959 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 02:34:48.994597 (+ 638us) client_negotiation.cc:240] Negotiation successful
0812 02:34:48.994831 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12461,"thread_start_us":12319,"threads_started":1}
I20260812 02:34:48.995438 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.972077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.12.8.193:43045)
0812 02:34:48.983250 (+ 11173us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:48.983262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:48.984690 (+ 1428us) server_negotiation.cc:407] Connection header received
0812 02:34:48.984869 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:48.984879 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:48.985366 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:48.985666 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:48.987060 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.988908 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.991623 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.993336 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.993500 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:48.993854 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:48.994724 (+ 870us) server_negotiation.cc:299] Negotiation successful
0812 02:34:48.994995 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10903,"thread_start_us":11210,"threads_started":1}
I20260812 02:34:49.001212 3205 tablet_service.cc:1467] Processing CreateTablet for tablet f1bac5c9c07946aca51a3476eae5c061 (DEFAULT_TABLE table=test-table [id=3f181a230d5d41f095da2609bf31f153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:49.001214 3030 tablet_service.cc:1467] Processing CreateTablet for tablet f1bac5c9c07946aca51a3476eae5c061 (DEFAULT_TABLE table=test-table [id=3f181a230d5d41f095da2609bf31f153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:49.003818 3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bac5c9c07946aca51a3476eae5c061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:49.003865 3030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bac5c9c07946aca51a3476eae5c061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:49.004024 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.973150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:44829 (local address 127.0.0.1:56342)
0812 02:34:48.989264 (+ 16114us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:48.989293 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:48.989441 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:48.990864 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:48.990870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:48.990915 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:48.991550 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.991566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.993715 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.993721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:48.998322 (+ 4601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:48.998351 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.001960 (+ 3609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.001979 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:49.002875 (+ 896us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.002929 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 02:34:49.003295 (+ 366us) client_negotiation.cc:240] Negotiation successful
0812 02:34:49.003533 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15918,"mutex_wait_us":1,"thread_start_us":16359,"threads_started":1}
I20260812 02:34:49.004093 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:48.973615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.12.8.195:44829)
0812 02:34:48.989494 (+ 15879us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:48.989512 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:48.989573 (+ 61us) server_negotiation.cc:407] Connection header received
0812 02:34:48.989728 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:48.989738 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:48.990142 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:48.990476 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:48.991840 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:48.993495 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:48.998720 (+ 5225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.000266 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.000474 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.002944 (+ 2470us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:49.003396 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 02:34:49.003631 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15236,"thread_start_us":15570,"threads_started":1}
I20260812 02:34:49.009338 3368 tablet_service.cc:1467] Processing CreateTablet for tablet f1bac5c9c07946aca51a3476eae5c061 (DEFAULT_TABLE table=test-table [id=3f181a230d5d41f095da2609bf31f153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:49.011337 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bac5c9c07946aca51a3476eae5c061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:49.022084 3478 tablet_bootstrap.cc:492] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Bootstrap starting.
I20260812 02:34:49.022775 3479 tablet_bootstrap.cc:492] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Bootstrap starting.
I20260812 02:34:49.028240 3479 tablet_bootstrap.cc:654] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:49.029349 3478 tablet_bootstrap.cc:654] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:49.029359 3481 tablet_bootstrap.cc:492] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Bootstrap starting.
I20260812 02:34:49.030828 3478 log.cc:824] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:49.030923 3479 log.cc:824] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:49.033424 3481 tablet_bootstrap.cc:654] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:49.034797 3481 log.cc:824] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:49.034946 3478 tablet_bootstrap.cc:492] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: No bootstrap required, opened a new log
I20260812 02:34:49.035259 3478 ts_tablet_manager.cc:1397] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20260812 02:34:49.035584 3479 tablet_bootstrap.cc:492] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: No bootstrap required, opened a new log
I20260812 02:34:49.035983 3479 ts_tablet_manager.cc:1397] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260812 02:34:49.039029 3481 tablet_bootstrap.cc:492] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: No bootstrap required, opened a new log
I20260812 02:34:49.039381 3481 ts_tablet_manager.cc:1397] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 02:34:49.045313 3479 raft_consensus.cc:348] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.045867 3479 raft_consensus.cc:374] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:49.045984 3479 raft_consensus.cc:729] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c4caaa44aa45b7b0f3b93b2e0ef556, State: Initialized, Role: FOLLOWER
I20260812 02:34:49.046762 3479 consensus_queue.cc:260] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.046370 3478 raft_consensus.cc:348] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.047300 3478 raft_consensus.cc:374] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:49.047447 3478 raft_consensus.cc:729] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b824bc4a340ebb03d623536d0fc80, State: Initialized, Role: FOLLOWER
I20260812 02:34:49.049321 3479 ts_tablet_manager.cc:1428] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260812 02:34:49.048345 3478 consensus_queue.cc:260] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.050956 3481 raft_consensus.cc:348] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.051513 3481 raft_consensus.cc:374] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:49.051610 3481 raft_consensus.cc:729] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31089272f8524fe2b98d851e5b9ff785, State: Initialized, Role: FOLLOWER
I20260812 02:34:49.051647 3281 heartbeater.cc:503] Master 127.12.8.254:41331 was elected leader, sending a full tablet report...
I20260812 02:34:49.052373 3481 consensus_queue.cc:260] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.052556 3478 ts_tablet_manager.cc:1428] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Time spent starting tablet: real 0.017s user 0.006s sys 0.008s
I20260812 02:34:49.054168 3451 heartbeater.cc:503] Master 127.12.8.254:41331 was elected leader, sending a full tablet report...
I20260812 02:34:49.054864 3481 ts_tablet_manager.cc:1428] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260812 02:34:49.125168 3282 tablet.cc:2369] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:34:49.126639 3123 tablet.cc:2369] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:34:49.146095 3452 tablet.cc:2369] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:49.275913 3488 raft_consensus.cc:482] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:49.276384 3488 raft_consensus.cc:504] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.279492 3488 leader_election.cc:288] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c4caaa44aa45b7b0f3b93b2e0ef556 (127.12.8.193:43045), 31089272f8524fe2b98d851e5b9ff785 (127.12.8.195:44829)
I20260812 02:34:49.287262 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:49.279827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:44829 (local address 127.12.8.194:56115)
0812 02:34:49.280710 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:49.280735 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:49.280937 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:49.281354 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:49.281359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:49.281387 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:49.281780 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:49.281791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.282898 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.282903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:49.284615 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:49.284637 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.286199 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.286206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:49.286534 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.286582 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 02:34:49.286732 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 02:34:49.286942 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":2,"thread_start_us":420,"threads_started":1}
I20260812 02:34:49.287349 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:49.280545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:56115 (local address 127.12.8.195:44829)
0812 02:34:49.280774 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:49.280780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:49.280826 (+ 46us) server_negotiation.cc:407] Connection header received
0812 02:34:49.281119 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:49.281123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:49.281224 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:49.281314 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:49.281926 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.282746 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.284866 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.285935 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.286146 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.286230 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:49.286803 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 02:34:49.286967 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 02:34:49.288699 3392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bac5c9c07946aca51a3476eae5c061" candidate_uuid: "c30b824bc4a340ebb03d623536d0fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31089272f8524fe2b98d851e5b9ff785" is_pre_election: true
I20260812 02:34:49.289417 3392 raft_consensus.cc:2454] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30b824bc4a340ebb03d623536d0fc80 in term 0.
I20260812 02:34:49.290354 3148 leader_election.cc:302] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31089272f8524fe2b98d851e5b9ff785, c30b824bc4a340ebb03d623536d0fc80; no voters:
I20260812 02:34:49.290874 3488 raft_consensus.cc:2781] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:49.291014 3488 raft_consensus.cc:482] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:49.291257 3488 raft_consensus.cc:3037] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:49.292384 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:49.280696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:60841 (local address 127.12.8.193:43045)
0812 02:34:49.280955 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:49.280961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:49.285683 (+ 4722us) server_negotiation.cc:407] Connection header received
0812 02:34:49.285886 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:49.285892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:49.286047 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:49.286149 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:49.286928 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.287958 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.290277 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.291299 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.291463 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.291520 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:49.291975 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 02:34:49.292117 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 02:34:49.292520 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:49.279933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:43045 (local address 127.12.8.194:60841)
0812 02:34:49.285532 (+ 5599us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:49.285556 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:49.285714 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:49.286173 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:49.286178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:49.286208 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:49.286734 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:49.286747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.288135 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.288140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:49.290038 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:49.290052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.291669 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.291674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:49.291811 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.291836 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:34:49.291919 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:34:49.292064 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5460,"mutex_wait_us":37,"thread_start_us":5170,"threads_started":1}
I20260812 02:34:49.293926 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bac5c9c07946aca51a3476eae5c061" candidate_uuid: "c30b824bc4a340ebb03d623536d0fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" is_pre_election: true
I20260812 02:34:49.294674 3054 raft_consensus.cc:2454] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30b824bc4a340ebb03d623536d0fc80 in term 0.
I20260812 02:34:49.294929 3488 raft_consensus.cc:504] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.295825 3488 leader_election.cc:288] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [CANDIDATE]: Term 1 election: Requested vote from peers a8c4caaa44aa45b7b0f3b93b2e0ef556 (127.12.8.193:43045), 31089272f8524fe2b98d851e5b9ff785 (127.12.8.195:44829)
I20260812 02:34:49.296435 3054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bac5c9c07946aca51a3476eae5c061" candidate_uuid: "c30b824bc4a340ebb03d623536d0fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556"
I20260812 02:34:49.296697 3054 raft_consensus.cc:3037] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:49.296691 3392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bac5c9c07946aca51a3476eae5c061" candidate_uuid: "c30b824bc4a340ebb03d623536d0fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31089272f8524fe2b98d851e5b9ff785"
I20260812 02:34:49.296947 3392 raft_consensus.cc:3037] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:49.300566 3054 raft_consensus.cc:2454] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b824bc4a340ebb03d623536d0fc80 in term 1.
I20260812 02:34:49.300566 3392 raft_consensus.cc:2454] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30b824bc4a340ebb03d623536d0fc80 in term 1.
I20260812 02:34:49.301249 3148 leader_election.cc:302] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31089272f8524fe2b98d851e5b9ff785, c30b824bc4a340ebb03d623536d0fc80; no voters:
I20260812 02:34:49.301687 3488 raft_consensus.cc:2781] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:49.302325 3488 raft_consensus.cc:686] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 1 LEADER]: Becoming Leader. State: Replica: c30b824bc4a340ebb03d623536d0fc80, State: Running, Role: LEADER
I20260812 02:34:49.302876 3488 consensus_queue.cc:237] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } }
I20260812 02:34:49.308815 2882 catalog_manager.cc:5495] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 reported cstate change: term changed from 0 to 1, leader changed from <none> to c30b824bc4a340ebb03d623536d0fc80 (127.12.8.194). New cstate: current_term: 1 leader_uuid: "c30b824bc4a340ebb03d623536d0fc80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30b824bc4a340ebb03d623536d0fc80" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 35405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 } health_report { overall_health: UNKNOWN } } }
I20260812 02:34:49.363603 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:49.358039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.12.8.194:35405)
0812 02:34:49.358328 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:49.358336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:49.358360 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:34:49.358482 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:49.358487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:49.358591 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:49.358735 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:49.359153 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.359881 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.361258 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.361697 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.361812 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.362426 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:49.362954 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:49.363141 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:49.363298 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 02:34:49.363346 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 02:34:49.374811 3054 raft_consensus.cc:1261] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 1 FOLLOWER]: Refusing update from remote peer c30b824bc4a340ebb03d623536d0fc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:49.374815 3392 raft_consensus.cc:1261] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 1 FOLLOWER]: Refusing update from remote peer c30b824bc4a340ebb03d623536d0fc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:49.376188 3493 consensus_queue.cc:1035] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31089272f8524fe2b98d851e5b9ff785" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:49.376714 3488 consensus_queue.cc:1035] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c4caaa44aa45b7b0f3b93b2e0ef556" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:49.394891 3496 mvcc.cc:205] Tried to move back new op lower bound from 7317512558063898624 to 7317512557788823552. Current Snapshot: MvccSnapshot[applied={T|T < 7317512558063898624}]
I20260812 02:34:49.415447 2882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:49.415800 2882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:49.420817 2882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 362dd0148de94ef59b9ae51f2cb955dc: Sending DeleteTablet for 3 replicas of tablet f1bac5c9c07946aca51a3476eae5c061
I20260812 02:34:49.422451 3030 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bac5c9c07946aca51a3476eae5c061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:49 UTC) from {username='slave'} at 127.0.0.1:48074
I20260812 02:34:49.422673 3205 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bac5c9c07946aca51a3476eae5c061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:49 UTC) from {username='slave'} at 127.0.0.1:56444
I20260812 02:34:49.423635 3507 tablet_replica.cc:330] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: stopping tablet replica
I20260812 02:34:49.424304 3507 raft_consensus.cc:2229] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:49.424458 3508 tablet_replica.cc:330] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: stopping tablet replica
I20260812 02:34:49.425053 3508 raft_consensus.cc:2229] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:34:49.425128 3368 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bac5c9c07946aca51a3476eae5c061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:49 UTC) from {username='slave'} at 127.0.0.1:56342
I20260812 02:34:49.425338 3507 raft_consensus.cc:2258] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:49.425942 3508 raft_consensus.cc:2258] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:49.427026 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 2957
I20260812 02:34:49.429260 3507 ts_tablet_manager.cc:1905] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:49.437567 3508 ts_tablet_manager.cc:1905] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:49.437995 3509 tablet_replica.cc:330] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: stopping tablet replica
I20260812 02:34:49.438498 3507 ts_tablet_manager.cc:1918] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:49.438578 3509 raft_consensus.cc:2229] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:49.438902 3507 log.cc:1196] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/f1bac5c9c07946aca51a3476eae5c061
I20260812 02:34:49.439188 3509 raft_consensus.cc:2258] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:49.440116 3507 ts_tablet_manager.cc:1939] T f1bac5c9c07946aca51a3476eae5c061 P a8c4caaa44aa45b7b0f3b93b2e0ef556: Deleting consensus metadata
I20260812 02:34:49.443100 2864 catalog_manager.cc:4841] TS a8c4caaa44aa45b7b0f3b93b2e0ef556 (127.12.8.193:43045): tablet f1bac5c9c07946aca51a3476eae5c061 (table test-table [id=3f181a230d5d41f095da2609bf31f153]) successfully deleted
I20260812 02:34:49.446522 3509 ts_tablet_manager.cc:1905] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:49.447641 3508 ts_tablet_manager.cc:1918] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:49.448913 3508 log.cc:1196] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/f1bac5c9c07946aca51a3476eae5c061
I20260812 02:34:49.449777 3508 ts_tablet_manager.cc:1939] T f1bac5c9c07946aca51a3476eae5c061 P c30b824bc4a340ebb03d623536d0fc80: Deleting consensus metadata
I20260812 02:34:49.452044 2867 catalog_manager.cc:4841] TS c30b824bc4a340ebb03d623536d0fc80 (127.12.8.194:35405): tablet f1bac5c9c07946aca51a3476eae5c061 (table test-table [id=3f181a230d5d41f095da2609bf31f153]) successfully deleted
I20260812 02:34:49.454046 3509 ts_tablet_manager.cc:1918] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:49.454263 3509 log.cc:1196] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/f1bac5c9c07946aca51a3476eae5c061
I20260812 02:34:49.454908 3509 ts_tablet_manager.cc:1939] T f1bac5c9c07946aca51a3476eae5c061 P 31089272f8524fe2b98d851e5b9ff785: Deleting consensus metadata
I20260812 02:34:49.457226 2867 catalog_manager.cc:4841] TS 31089272f8524fe2b98d851e5b9ff785 (127.12.8.195:44829): tablet f1bac5c9c07946aca51a3476eae5c061 (table test-table [id=3f181a230d5d41f095da2609bf31f153]) successfully deleted
I20260812 02:34:49.458354 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:49.428487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.12.8.193:43045)
0812 02:34:49.428698 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:49.428705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:49.448604 (+ 19899us) server_negotiation.cc:407] Connection header received
0812 02:34:49.448823 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:49.448829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:49.448967 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:49.449126 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:49.449946 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.451091 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.454043 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.454744 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.454914 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.456359 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:49.456371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:49.456395 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:49.457410 (+ 1015us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:49.457488 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:49.457517 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:49.457622 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:49.457794 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 02:34:49.457914 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:34:49.459100 3114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:49.608856 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 2957
I20260812 02:34:49.630581 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 3128
I20260812 02:34:49.638842 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:49.631553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.12.8.194:35405)
0812 02:34:49.631764 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:49.631771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:49.631793 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:34:49.631862 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:49.631868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:49.631977 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:49.632083 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:49.632515 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.633690 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.635045 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.635722 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.635854 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.636975 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:49.636986 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:49.637008 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:49.637947 (+ 939us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:49.638014 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:49.638047 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:49.638163 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:49.638330 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 02:34:49.638450 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:34:49.639662 3276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:49.757355 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 3128
I20260812 02:34:49.783051 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 3285
I20260812 02:34:49.789901 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:49.783949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.12.8.195:44829)
0812 02:34:49.784119 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:49.784124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:49.784142 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:34:49.784243 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:49.784247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:49.784349 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:49.784431 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:49.784811 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.785608 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.786949 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:49.787362 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:49.787453 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:49.788530 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:49.788539 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:49.788553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:49.789117 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:49.789166 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:49.789225 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:49.789330 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:49.789458 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 02:34:49.789552 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:34:49.790576 3446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:49.879181 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 3285
I20260812 02:34:49.897665 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 2846
I20260812 02:34:49.898779 2923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:50.009699 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 2846
2026-08-12T02:34:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3467 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:34:50 dist-test-slave-vq2q krb5kdc[3521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:34:50 dist-test-slave-vq2q krb5kdc[3521](info): set up 2 sockets
Aug 12 02:34:50 dist-test-slave-vq2q krb5kdc[3521](info): commencing operation
krb5kdc: starting...
W20260812 02:34:51.158187 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:34:51 dist-test-slave-vq2q krb5kdc[3521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:34:51.488299 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:32947
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:32947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:51.843817 3536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:51.844125 3536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:51.844215 3536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:51.851759 3536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:51.851898 3536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:51.851946 3536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:51.852005 3536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:51.852049 3536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:51.862756 3536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:32947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:32947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:51.864152 3536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:51.865517 3536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:51.877736 3544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:51.877842 3542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:51.877777 3541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:51.879010 3536 server_base.cc:1034] running on GCE node
I20260812 02:34:51.879846 3536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:51.881440 3536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:51.882663 3536 hybrid_clock.cc:648] HybridClock initialized: now 1786502091882611 us; error 41 us; skew 500 ppm
I20260812 02:34:51.883066 3536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:51 dist-test-slave-vq2q krb5kdc[3521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:51.886307 3536 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:34:51.887815 3536 webserver.cc:460] Webserver started at http://127.12.8.254:45603/ using document root <none> and password file <none>
I20260812 02:34:51.888499 3536 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:51.888612 3536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:51.888882 3536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:51.892052 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "40a18975b872419b83c62f0e72d189b6"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-vq2q"
server_key: "f49518a00941ac6eee783f83ee13eb20"
server_key_iv: "82e341f7b361eb2b60242db9fcdf514f"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.892795 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "40a18975b872419b83c62f0e72d189b6"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-vq2q"
server_key: "f49518a00941ac6eee783f83ee13eb20"
server_key_iv: "82e341f7b361eb2b60242db9fcdf514f"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.898284 3536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 02:34:51.901876 3556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:51.903335 3536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:34:51.903517 3536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "40a18975b872419b83c62f0e72d189b6"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-vq2q"
server_key: "f49518a00941ac6eee783f83ee13eb20"
server_key_iv: "82e341f7b361eb2b60242db9fcdf514f"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.903718 3536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:51.939204 3536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:51.943773 3536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:51.944067 3536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:51.962359 3536 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:32947
I20260812 02:34:51.962405 3614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:32947 every 8 connection(s)
I20260812 02:34:51.963847 3536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:51.967818 3615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:51.972846 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 3536
I20260812 02:34:51.973057 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:34:51.973563 12323 external_mini_cluster.cc:1428] Setting key debf328a236b8644c45215a9c439c10a
I20260812 02:34:51.979012 3615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6: Bootstrap starting.
I20260812 02:34:51.982859 3615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:51.984221 3615 log.cc:824] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:51.987118 3615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6: No bootstrap required, opened a new log
Aug 12 02:34:51 dist-test-slave-vq2q krb5kdc[3521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:51.997534 3615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a18975b872419b83c62f0e72d189b6" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 32947 } }
I20260812 02:34:51.998024 3615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:51.998135 3615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a18975b872419b83c62f0e72d189b6, State: Initialized, Role: FOLLOWER
I20260812 02:34:51.998379 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:51.975394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.12.8.254:32947)
0812 02:34:51.976290 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:51.976302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:51.976347 (+ 45us) server_negotiation.cc:407] Connection header received
0812 02:34:51.978142 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:51.978165 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:51.978555 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:51.978873 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:51.980853 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:51.982227 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:51.984020 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:51.984759 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:51.984961 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:51.989251 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:51.989263 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:51.989282 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:51.989343 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:51.993452 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:51.994227 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:51.994240 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:51.994249 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:51.994474 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:51.994849 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:51.994854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:51.994858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:51.995699 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:51.996028 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:51.996437 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 02:34:51.996911 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":384,"threads_started":1}
I20260812 02:34:51.998828 3615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a18975b872419b83c62f0e72d189b6" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 32947 } }
I20260812 02:34:51.999075 3615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:51.999218 3615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:51.999374 3615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:52.002285 3615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a18975b872419b83c62f0e72d189b6" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 32947 } }
I20260812 02:34:52.002785 3615 leader_election.cc:302] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40a18975b872419b83c62f0e72d189b6; no voters:
I20260812 02:34:52.003412 3615 leader_election.cc:288] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:52.003587 3621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:52.004377 3621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [term 1 LEADER]: Becoming Leader. State: Replica: 40a18975b872419b83c62f0e72d189b6, State: Running, Role: LEADER
I20260812 02:34:52.004984 3621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a18975b872419b83c62f0e72d189b6" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 32947 } }
I20260812 02:34:52.005874 3615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:52.008096 3623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40a18975b872419b83c62f0e72d189b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a18975b872419b83c62f0e72d189b6" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 32947 } } }
I20260812 02:34:52.008476 3623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:52.009100 3622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40a18975b872419b83c62f0e72d189b6. Latest consensus state: current_term: 1 leader_uuid: "40a18975b872419b83c62f0e72d189b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a18975b872419b83c62f0e72d189b6" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 32947 } } }
I20260812 02:34:52.009574 3622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:52.009323 3629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:52.015338 3629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:52.024312 3629 catalog_manager.cc:1347] Generated new cluster ID: 7bcb44d13f1f40eeac0a4f4410a91583
I20260812 02:34:52.024436 3629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:52.049458 3629 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:52.050854 3629 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:52.057192 3629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6: Generated new TSK 0
I20260812 02:34:52.057998 3629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
Entry for principal kudu/127.12.8.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab.
I20260812 02:34:52.153071 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:32947
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:52.558761 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:52.559157 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:52.559332 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:52.567296 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:52.567444 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:52.567710 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:34:52.578652 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.193.keytab
--principal=kudu/127.12.8.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:32947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:52.580098 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:52.581812 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:52.597128 3658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:52.597363 3657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:52.597363 3660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:52.598891 3650 server_base.cc:1034] running on GCE node
I20260812 02:34:52.599709 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:52.600888 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:52.602267 3650 hybrid_clock.cc:648] HybridClock initialized: now 1786502092602216 us; error 84 us; skew 500 ppm
I20260812 02:34:52.602681 3650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:52 dist-test-slave-vq2q krb5kdc[3521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502092, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:52.606490 3650 init.cc:377] Logged in from keytab as kudu/127.12.8.193@KRBTEST.COM (short username kudu)
I20260812 02:34:52.608434 3650 webserver.cc:460] Webserver started at http://127.12.8.193:40863/ using document root <none> and password file <none>
I20260812 02:34:52.609275 3650 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:52.609418 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:52.609772 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:52.613171 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "4acf66e9d707492a87f777bee09de004"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-vq2q"
server_key: "cb6f834cc984b9023e442f2d56fb077d"
server_key_iv: "e965a6828b8249df1eb3339cb27cfffb"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.614094 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "4acf66e9d707492a87f777bee09de004"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-vq2q"
server_key: "cb6f834cc984b9023e442f2d56fb077d"
server_key_iv: "e965a6828b8249df1eb3339cb27cfffb"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.620548 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 02:34:52.625337 3673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.627342 3650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:52.627568 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "4acf66e9d707492a87f777bee09de004"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-vq2q"
server_key: "cb6f834cc984b9023e442f2d56fb077d"
server_key_iv: "e965a6828b8249df1eb3339cb27cfffb"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.627810 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:52.647296 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:52.652439 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:52.652750 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:52.654042 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:52.655820 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:52.655905 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.656036 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:52.656090 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.695119 3650 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:42851
I20260812 02:34:52.695212 3806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:42851 every 8 connection(s)
I20260812 02:34:52.696593 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:52.700019 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 3650
I20260812 02:34:52.700331 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:34:52.701027 12323 external_mini_cluster.cc:1428] Setting key e145a966e3ae9328146e05077cd12d57
Aug 12 02:34:52 dist-test-slave-vq2q krb5kdc[3521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502092, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.193@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:52.727439 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.700389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:38789 (local address 127.12.8.254:32947)
0812 02:34:52.701098 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.701107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.702521 (+ 1414us) server_negotiation.cc:407] Connection header received
0812 02:34:52.705674 (+ 3153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.705679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.705815 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.705941 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:52.709068 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.709967 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.713057 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.713620 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.713773 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.719226 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:52.719236 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:52.719242 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:52.719310 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:52.723095 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:52.724231 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:52.724238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:52.724242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:52.724486 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:52.725002 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:52.725007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:52.725009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:52.725460 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:52.725625 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.726776 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.727113 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":310,"threads_started":1}
I20260812 02:34:52.728824 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.701030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:32947 (local address 127.12.8.193:38789)
0812 02:34:52.702220 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.702316 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:52.705068 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:52.706444 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:52.706463 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:52.707600 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:52.708744 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.708786 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.710220 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.710226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.712785 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.712811 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.713979 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.713988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.714381 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.715352 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:52.715398 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:52.718724 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:52.723390 (+ 4666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:52.723407 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:52.723425 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:52.724053 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:52.724628 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:52.724632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:52.724634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:52.724868 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:52.725655 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:52.725662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:52.726240 (+ 578us) client_negotiation.cc:769] Sending connection context
0812 02:34:52.726643 (+ 403us) client_negotiation.cc:240] Negotiation successful
0812 02:34:52.727334 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":415,"threads_started":1}
I20260812 02:34:52.731060 3807 heartbeater.cc:347] Connected to a master server at 127.12.8.254:32947
I20260812 02:34:52.731554 3807 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:52.732676 3807 heartbeater.cc:511] Master 127.12.8.254:32947 requested a full tablet report, sending...
I20260812 02:34:52.735790 3576 ts_manager.cc:194] Registered new tserver with Master: 4acf66e9d707492a87f777bee09de004 (127.12.8.193:42851)
I20260812 02:34:52.737740 3576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.193@KRBTEST.COM'} at 127.12.8.193:38789
WARNING: no policy specified for kudu/127.12.8.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
Entry for principal kudu/127.12.8.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab.
I20260812 02:34:52.794211 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:32947
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:53.161099 3816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:53.161465 3816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:53.161598 3816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:53.169066 3816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:53.169235 3816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:53.169394 3816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:34:53.179646 3816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.194.keytab
--principal=kudu/127.12.8.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:32947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:53.181056 3816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:53.182451 3816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:53.196902 3822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.197000 3823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.197394 3827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:53.198346 3816 server_base.cc:1034] running on GCE node
I20260812 02:34:53.199393 3816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:53.200677 3816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:53.201951 3816 hybrid_clock.cc:648] HybridClock initialized: now 1786502093201888 us; error 51 us; skew 500 ppm
I20260812 02:34:53.202466 3816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:53 dist-test-slave-vq2q krb5kdc[3521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:53.206874 3816 init.cc:377] Logged in from keytab as kudu/127.12.8.194@KRBTEST.COM (short username kudu)
I20260812 02:34:53.208957 3816 webserver.cc:460] Webserver started at http://127.12.8.194:38793/ using document root <none> and password file <none>
I20260812 02:34:53.209937 3816 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:53.210089 3816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:53.210469 3816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:53.214433 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "9dad9550591c434ab9d6c937f3d768e1"
format_stamp: "Formatted at 2026-08-12 02:34:53 on dist-test-slave-vq2q"
server_key: "43623e4be65d36c7f46ca4f77f32d1ea"
server_key_iv: "691427bc729f0e511b5392175a879627"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.215404 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "9dad9550591c434ab9d6c937f3d768e1"
format_stamp: "Formatted at 2026-08-12 02:34:53 on dist-test-slave-vq2q"
server_key: "43623e4be65d36c7f46ca4f77f32d1ea"
server_key_iv: "691427bc729f0e511b5392175a879627"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.222486 3816 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 02:34:53.227592 3834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.229264 3816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:34:53.229444 3816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "9dad9550591c434ab9d6c937f3d768e1"
format_stamp: "Formatted at 2026-08-12 02:34:53 on dist-test-slave-vq2q"
server_key: "43623e4be65d36c7f46ca4f77f32d1ea"
server_key_iv: "691427bc729f0e511b5392175a879627"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.229682 3816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:53.242295 3816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:53.246658 3816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:53.246914 3816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:53.247987 3816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:53.249557 3816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:53.249639 3816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.249763 3816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:53.249816 3816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.285619 3816 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:42723
I20260812 02:34:53.285766 3967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:42723 every 8 connection(s)
I20260812 02:34:53.287186 3816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:53.289839 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 3816
I20260812 02:34:53.290112 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:34:53.290766 12323 external_mini_cluster.cc:1428] Setting key 69481461cc771cedde468edd5518fbc0
Aug 12 02:34:53 dist-test-slave-vq2q krb5kdc[3521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.194@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:53.311229 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.291807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:42791 (local address 127.12.8.254:32947)
0812 02:34:53.292455 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.292462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.293510 (+ 1048us) server_negotiation.cc:407] Connection header received
0812 02:34:53.295655 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.295661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.295750 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.295847 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:53.297811 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.298645 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.300380 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.300863 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.300981 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.304671 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:53.304676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:53.304680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:53.304719 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:53.307917 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.308657 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.308661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.308663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.308804 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.309282 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.309287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.309290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.309598 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:53.309740 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.310492 (+ 752us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.310729 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":316,"threads_started":1}
I20260812 02:34:53.312141 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.292238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:32947 (local address 127.12.8.194:42791)
0812 02:34:53.293283 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.293354 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.295342 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.296123 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.296131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.296711 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:53.297594 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.297622 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.298809 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.298813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.300206 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.300227 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.301115 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.301123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.301411 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.302107 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:53.302132 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:53.304412 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:53.308061 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.308071 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.308086 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.308526 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.308910 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.308914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.308917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.309139 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.309727 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:53.309735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:53.310092 (+ 357us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.310331 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.310868 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":374,"threads_started":1}
I20260812 02:34:53.313860 3969 heartbeater.cc:347] Connected to a master server at 127.12.8.254:32947
I20260812 02:34:53.314177 3969 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:53.314927 3969 heartbeater.cc:511] Master 127.12.8.254:32947 requested a full tablet report, sending...
I20260812 02:34:53.316766 3576 ts_manager.cc:194] Registered new tserver with Master: 9dad9550591c434ab9d6c937f3d768e1 (127.12.8.194:42723)
I20260812 02:34:53.317878 3576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.194@KRBTEST.COM'} at 127.12.8.194:42791
WARNING: no policy specified for kudu/127.12.8.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
Entry for principal kudu/127.12.8.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab.
I20260812 02:34:53.379176 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:32947
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:53.742070 3807 heartbeater.cc:503] Master 127.12.8.254:32947 was elected leader, sending a full tablet report...
W20260812 02:34:53.743908 3976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:53.744382 3976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:53.744606 3976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:53.752313 3976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:53.752475 3976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:53.752640 3976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:34:53.762990 3976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.195.keytab
--principal=kudu/127.12.8.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:32947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:53.764353 3976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:53.765774 3976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:53.781514 3984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.781728 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.782145 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:53.783224 3976 server_base.cc:1034] running on GCE node
I20260812 02:34:53.783975 3976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:53.785115 3976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:53.786386 3976 hybrid_clock.cc:648] HybridClock initialized: now 1786502093786322 us; error 65 us; skew 500 ppm
I20260812 02:34:53.786757 3976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:34:53 dist-test-slave-vq2q krb5kdc[3521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:53.790339 3976 init.cc:377] Logged in from keytab as kudu/127.12.8.195@KRBTEST.COM (short username kudu)
I20260812 02:34:53.792229 3976 webserver.cc:460] Webserver started at http://127.12.8.195:40365/ using document root <none> and password file <none>
I20260812 02:34:53.792963 3976 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:53.793088 3976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:53.793483 3976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:53.796545 3976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "479d1314317d4b4e987303c26ab199db"
format_stamp: "Formatted at 2026-08-12 02:34:53 on dist-test-slave-vq2q"
server_key: "b9e859d705e3720bc1ee18c3f5b0727b"
server_key_iv: "07d3fdd35a7592701d02239e43004a21"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.797381 3976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "479d1314317d4b4e987303c26ab199db"
format_stamp: "Formatted at 2026-08-12 02:34:53 on dist-test-slave-vq2q"
server_key: "b9e859d705e3720bc1ee18c3f5b0727b"
server_key_iv: "07d3fdd35a7592701d02239e43004a21"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.803791 3976 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:34:53.808249 3995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.809959 3976 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:34:53.810148 3976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "479d1314317d4b4e987303c26ab199db"
format_stamp: "Formatted at 2026-08-12 02:34:53 on dist-test-slave-vq2q"
server_key: "b9e859d705e3720bc1ee18c3f5b0727b"
server_key_iv: "07d3fdd35a7592701d02239e43004a21"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.810366 3976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:53.834074 3976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:53.838794 3976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:53.839094 3976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:53.840235 3976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:53.841780 3976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:53.841864 3976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.841989 3976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:53.842043 3976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.878841 3976 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:41989
I20260812 02:34:53.878934 4129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:41989 every 8 connection(s)
I20260812 02:34:53.880208 3976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:53.883713 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 3976
I20260812 02:34:53.883965 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:34:53.884620 12323 external_mini_cluster.cc:1428] Setting key 93c273fd2fc95821ebc432e9df9a5851
Aug 12 02:34:53 dist-test-slave-vq2q krb5kdc[3521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.195@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:34:53.903959 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.883502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:51393 (local address 127.12.8.254:32947)
0812 02:34:53.884225 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.884233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.885146 (+ 913us) server_negotiation.cc:407] Connection header received
0812 02:34:53.887499 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.887503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.887634 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.887747 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:53.889572 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.890499 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.892336 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.892839 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.892968 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.896927 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:53.896934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:53.896938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:53.896989 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:53.900316 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.901303 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.901310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.901314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.901518 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.902036 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.902040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.902043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.902417 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:53.902557 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.903382 (+ 825us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.903615 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":320,"threads_started":1}
I20260812 02:34:53.904879 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.884015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:32947 (local address 127.12.8.195:51393)
0812 02:34:53.884943 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.884993 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.887137 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.888060 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.888071 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.888704 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:53.889361 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.889382 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.890787 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.890800 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.892133 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.892148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.893078 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.893083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.893372 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.894139 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:53.894167 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:53.896621 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:53.900496 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.900504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.900519 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.901129 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.901652 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.901656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.901659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.901898 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.902600 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:53.902606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:53.902988 (+ 382us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.903240 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.903672 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":388,"threads_started":1}
I20260812 02:34:53.906541 4130 heartbeater.cc:347] Connected to a master server at 127.12.8.254:32947
I20260812 02:34:53.907001 4130 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:53.907959 4130 heartbeater.cc:511] Master 127.12.8.254:32947 requested a full tablet report, sending...
I20260812 02:34:53.909832 3576 ts_manager.cc:194] Registered new tserver with Master: 479d1314317d4b4e987303c26ab199db (127.12.8.195:41989)
I20260812 02:34:53.910833 3576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.195@KRBTEST.COM'} at 127.12.8.195:51393
I20260812 02:34:53.914539 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:53.934466 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.919449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.12.8.254:32947)
0812 02:34:53.919677 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.919683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.920016 (+ 333us) server_negotiation.cc:407] Connection header received
0812 02:34:53.920179 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.920183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.920275 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.920401 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:53.922119 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.923061 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.924478 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.924978 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.925142 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.927861 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:53.927867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:53.927872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:53.927929 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:53.931109 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.931866 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.931870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.931873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.932046 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.932512 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.932517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.932519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.933048 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:53.933404 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.933868 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.934171 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:34:53.939055 3576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:53.947080 3576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:53.974134 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.962664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.12.8.193:42851)
0812 02:34:53.963383 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.963390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.964023 (+ 633us) server_negotiation.cc:407] Connection header received
0812 02:34:53.964325 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.964332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.964557 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.964747 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:53.965989 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.967705 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.970775 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.972393 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.972511 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.972775 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.973524 (+ 749us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.973728 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":399,"threads_started":1}
I20260812 02:34:53.975386 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.962456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:42851 (local address 127.0.0.1:57614)
0812 02:34:53.963886 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.963917 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.964101 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.965248 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.965256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.965290 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:53.965738 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.965754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.967921 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.967927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.970572 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.970591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.972994 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.973001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.973248 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.973307 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.974667 (+ 1360us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.974937 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"thread_start_us":1070,"threads_started":1}
I20260812 02:34:53.986122 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.964854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.12.8.195:41989)
0812 02:34:53.977758 (+ 12904us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.977767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.977811 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:34:53.977916 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.977925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.978203 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.978858 (+ 655us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:53.979077 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.980054 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.982238 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.984828 (+ 2590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.985085 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.985360 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.985575 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.985773 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12732,"thread_start_us":12560,"threads_started":1}
I20260812 02:34:53.986205 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.964550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:41989 (local address 127.0.0.1:38272)
0812 02:34:53.975867 (+ 11317us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.975889 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.976005 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.978422 (+ 2417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.978427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.978453 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:53.978700 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.978713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.980221 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.980227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.982030 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.982050 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.985009 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.985020 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.985353 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.985399 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.985509 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.985708 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11195,"mutex_wait_us":22,"thread_start_us":15759,"threads_started":1}
I20260812 02:34:53.985945 3728 tablet_service.cc:1467] Processing CreateTablet for tablet dd67a645b78141899f72baa056edd4ce (DEFAULT_TABLE table=test-table [id=e50c52afeb7b4f9fadeac0340141901b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:53.987848 3728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd67a645b78141899f72baa056edd4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:53.990840 4049 tablet_service.cc:1467] Processing CreateTablet for tablet dd67a645b78141899f72baa056edd4ce (DEFAULT_TABLE table=test-table [id=e50c52afeb7b4f9fadeac0340141901b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:53.992923 4049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd67a645b78141899f72baa056edd4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:53.994867 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.980899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42723 (local address 127.0.0.1:57968)
0812 02:34:53.983611 (+ 2712us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.983634 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.983806 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.984919 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.984925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.984953 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:53.986671 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.986686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.988659 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.988664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.990868 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.990888 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.993824 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.993833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.994088 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.994136 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.994273 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.994455 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2545,"mutex_wait_us":3}
I20260812 02:34:53.994935 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.981207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.12.8.194:42723)
0812 02:34:53.983153 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.983165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.983954 (+ 789us) server_negotiation.cc:407] Connection header received
0812 02:34:53.984108 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.984118 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.984423 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.984636 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:53.986864 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.988464 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.991410 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.993522 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.993707 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.993911 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.994381 (+ 470us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.994576 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1726,"thread_start_us":2002,"threads_started":1}
I20260812 02:34:53.999907 3889 tablet_service.cc:1467] Processing CreateTablet for tablet dd67a645b78141899f72baa056edd4ce (DEFAULT_TABLE table=test-table [id=e50c52afeb7b4f9fadeac0340141901b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:54.001652 3889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd67a645b78141899f72baa056edd4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:54.003636 4155 tablet_bootstrap.cc:492] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Bootstrap starting.
I20260812 02:34:54.008823 4155 tablet_bootstrap.cc:654] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:54.010861 4155 log.cc:824] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:54.011809 4156 tablet_bootstrap.cc:492] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Bootstrap starting.
I20260812 02:34:54.016180 4156 tablet_bootstrap.cc:654] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:54.016348 4158 tablet_bootstrap.cc:492] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Bootstrap starting.
I20260812 02:34:54.017514 4156 log.cc:824] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:54.020555 4156 tablet_bootstrap.cc:492] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: No bootstrap required, opened a new log
I20260812 02:34:54.020833 4156 ts_tablet_manager.cc:1397] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 02:34:54.021390 4158 tablet_bootstrap.cc:654] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:54.023293 4155 tablet_bootstrap.cc:492] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: No bootstrap required, opened a new log
I20260812 02:34:54.023363 4158 log.cc:824] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:54.023694 4155 ts_tablet_manager.cc:1397] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20260812 02:34:54.028370 4158 tablet_bootstrap.cc:492] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: No bootstrap required, opened a new log
I20260812 02:34:54.028766 4158 ts_tablet_manager.cc:1397] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260812 02:34:54.029296 4156 raft_consensus.cc:348] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
I20260812 02:34:54.029701 4156 raft_consensus.cc:374] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:54.029807 4156 raft_consensus.cc:729] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479d1314317d4b4e987303c26ab199db, State: Initialized, Role: FOLLOWER
I20260812 02:34:54.030408 4156 consensus_queue.cc:260] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
I20260812 02:34:54.031708 4130 heartbeater.cc:503] Master 127.12.8.254:32947 was elected leader, sending a full tablet report...
I20260812 02:34:54.032284 4156 ts_tablet_manager.cc:1428] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260812 02:34:54.037919 4155 raft_consensus.cc:348] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
I20260812 02:34:54.038573 4155 raft_consensus.cc:374] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:54.038695 4155 raft_consensus.cc:729] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acf66e9d707492a87f777bee09de004, State: Initialized, Role: FOLLOWER
I20260812 02:34:54.039552 4155 consensus_queue.cc:260] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
I20260812 02:34:54.040465 4158 raft_consensus.cc:348] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
I20260812 02:34:54.041033 4158 raft_consensus.cc:374] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:54.041216 4158 raft_consensus.cc:729] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dad9550591c434ab9d6c937f3d768e1, State: Initialized, Role: FOLLOWER
I20260812 02:34:54.041918 4155 ts_tablet_manager.cc:1428] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260812 02:34:54.041976 4158 consensus_queue.cc:260] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
I20260812 02:34:54.043576 3969 heartbeater.cc:503] Master 127.12.8.254:32947 was elected leader, sending a full tablet report...
I20260812 02:34:54.046025 4158 ts_tablet_manager.cc:1428] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Time spent starting tablet: real 0.017s user 0.006s sys 0.011s
W20260812 02:34:54.132002 4131 tablet.cc:2369] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:54.172339 4162 raft_consensus.cc:482] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:54.172768 4162 raft_consensus.cc:504] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
I20260812 02:34:54.175055 4162 leader_election.cc:288] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4acf66e9d707492a87f777bee09de004 (127.12.8.193:42851), 9dad9550591c434ab9d6c937f3d768e1 (127.12.8.194:42723)
I20260812 02:34:54.182030 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.175570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:49401 (local address 127.12.8.193:42851)
0812 02:34:54.175894 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:54.175901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:54.175932 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:34:54.176195 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:54.176202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:54.176308 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:54.176431 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:54.177136 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.178028 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.180054 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.180964 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.181148 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:54.181277 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:54.181571 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 02:34:54.181731 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 02:34:54.182037 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.175511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:42851 (local address 127.12.8.195:49401)
0812 02:34:54.175769 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.175791 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:54.175983 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:54.176546 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:54.176553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:54.176608 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:54.176853 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:54.176866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.178225 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.178230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:54.179848 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:54.179867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.181149 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.181155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:54.181383 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:54.181412 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:34:54.181513 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 02:34:54.181709 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 02:34:54.182256 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.176216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:46619 (local address 127.12.8.194:42723)
0812 02:34:54.176419 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:54.176426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:54.176997 (+ 571us) server_negotiation.cc:407] Connection header received
0812 02:34:54.177152 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:54.177157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:54.177265 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:54.177345 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:54.177844 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.178582 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.180472 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.181291 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.181437 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:54.181515 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:54.181837 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 02:34:54.181970 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:34:54.182830 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.176024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:42723 (local address 127.12.8.195:46619)
0812 02:34:54.176857 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.176874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:54.177000 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:54.177373 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:54.177378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:54.177401 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:54.177697 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:54.177706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.178744 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.178748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:54.180320 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:54.180333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.181505 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.181512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:54.181664 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:54.181691 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:34:54.181814 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 02:34:54.181972 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":6,"thread_start_us":493,"threads_started":1}
I20260812 02:34:54.183403 3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd67a645b78141899f72baa056edd4ce" candidate_uuid: "479d1314317d4b4e987303c26ab199db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acf66e9d707492a87f777bee09de004" is_pre_election: true
I20260812 02:34:54.183904 3910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd67a645b78141899f72baa056edd4ce" candidate_uuid: "479d1314317d4b4e987303c26ab199db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dad9550591c434ab9d6c937f3d768e1" is_pre_election: true
I20260812 02:34:54.184106 3751 raft_consensus.cc:2454] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479d1314317d4b4e987303c26ab199db in term 0.
I20260812 02:34:54.184466 3910 raft_consensus.cc:2454] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479d1314317d4b4e987303c26ab199db in term 0.
I20260812 02:34:54.184950 3996 leader_election.cc:302] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479d1314317d4b4e987303c26ab199db, 4acf66e9d707492a87f777bee09de004; no voters:
I20260812 02:34:54.185451 4162 raft_consensus.cc:2781] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:54.185645 4162 raft_consensus.cc:482] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:54.185773 4162 raft_consensus.cc:3037] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:54.188508 4162 raft_consensus.cc:504] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
I20260812 02:34:54.189400 4162 leader_election.cc:288] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [CANDIDATE]: Term 1 election: Requested vote from peers 4acf66e9d707492a87f777bee09de004 (127.12.8.193:42851), 9dad9550591c434ab9d6c937f3d768e1 (127.12.8.194:42723)
I20260812 02:34:54.190126 3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd67a645b78141899f72baa056edd4ce" candidate_uuid: "479d1314317d4b4e987303c26ab199db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acf66e9d707492a87f777bee09de004"
I20260812 02:34:54.190225 3910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd67a645b78141899f72baa056edd4ce" candidate_uuid: "479d1314317d4b4e987303c26ab199db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dad9550591c434ab9d6c937f3d768e1"
I20260812 02:34:54.190445 3910 raft_consensus.cc:3037] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:54.190467 3751 raft_consensus.cc:3037] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:54.194154 3751 raft_consensus.cc:2454] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479d1314317d4b4e987303c26ab199db in term 1.
I20260812 02:34:54.194154 3910 raft_consensus.cc:2454] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479d1314317d4b4e987303c26ab199db in term 1.
I20260812 02:34:54.194806 3996 leader_election.cc:302] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479d1314317d4b4e987303c26ab199db, 4acf66e9d707492a87f777bee09de004; no voters:
I20260812 02:34:54.195220 4162 raft_consensus.cc:2781] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:54.195792 4162 raft_consensus.cc:686] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 1 LEADER]: Becoming Leader. State: Replica: 479d1314317d4b4e987303c26ab199db, State: Running, Role: LEADER
I20260812 02:34:54.196415 4162 consensus_queue.cc:237] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } }
W20260812 02:34:54.199905 3809 tablet.cc:2369] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:54.202073 3575 catalog_manager.cc:5495] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db reported cstate change: term changed from 0 to 1, leader changed from <none> to 479d1314317d4b4e987303c26ab199db (127.12.8.195). New cstate: current_term: 1 leader_uuid: "479d1314317d4b4e987303c26ab199db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479d1314317d4b4e987303c26ab199db" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 41989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 } health_report { overall_health: UNKNOWN } } }
I20260812 02:34:54.273634 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.268083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.12.8.195:41989)
0812 02:34:54.268343 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:54.268349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:54.268364 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:34:54.268458 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:54.268461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:54.268544 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:54.268681 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:54.269078 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.269874 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.271348 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.271737 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.271848 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:54.272419 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:54.272965 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:54.273109 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:54.273262 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 02:34:54.273354 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 02:34:54.283416 3751 raft_consensus.cc:1261] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 1 FOLLOWER]: Refusing update from remote peer 479d1314317d4b4e987303c26ab199db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:54.283474 3910 raft_consensus.cc:1261] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 1 FOLLOWER]: Refusing update from remote peer 479d1314317d4b4e987303c26ab199db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:54.284641 4162 consensus_queue.cc:1035] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acf66e9d707492a87f777bee09de004" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:54.285158 4166 consensus_queue.cc:1035] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dad9550591c434ab9d6c937f3d768e1" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:34:54.289669 3971 tablet.cc:2369] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:54.302740 4170 mvcc.cc:205] Tried to move back new op lower bound from 7317512578171551744 to 7317512577833332736. Current Snapshot: MvccSnapshot[applied={T|T < 7317512578171551744}]
I20260812 02:34:54.313669 4172 mvcc.cc:205] Tried to move back new op lower bound from 7317512578171551744 to 7317512577833332736. Current Snapshot: MvccSnapshot[applied={T|T < 7317512578171551744}]
I20260812 02:34:54.322098 3576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:54.322499 3576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:54.328272 3576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40a18975b872419b83c62f0e72d189b6: Sending DeleteTablet for 3 replicas of tablet dd67a645b78141899f72baa056edd4ce
I20260812 02:34:54.329625 3889 tablet_service.cc:1514] Processing DeleteTablet for tablet dd67a645b78141899f72baa056edd4ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:54 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:57968
I20260812 02:34:54.329813 3728 tablet_service.cc:1514] Processing DeleteTablet for tablet dd67a645b78141899f72baa056edd4ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:54 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:57614
I20260812 02:34:54.331043 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 3650
I20260812 02:34:54.331204 4049 tablet_service.cc:1514] Processing DeleteTablet for tablet dd67a645b78141899f72baa056edd4ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:54 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:38272
I20260812 02:34:54.332263 3888 tablet_service.cc:1514] Processing DeleteTablet for tablet dd67a645b78141899f72baa056edd4ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:54 UTC) from {username='kudu', principal='kudu/127.12.8.254@KRBTEST.COM'} at 127.0.0.1:57968
I20260812 02:34:54.343581 4181 tablet_replica.cc:330] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: stopping tablet replica
I20260812 02:34:54.344259 4181 raft_consensus.cc:2229] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:54.344990 4181 raft_consensus.cc:2258] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:54.345865 4183 tablet_replica.cc:330] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: stopping tablet replica
I20260812 02:34:54.346459 4183 raft_consensus.cc:2229] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:34:54.347296 4183 raft_consensus.cc:2258] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:54.348225 4181 ts_tablet_manager.cc:1905] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:54.349679 4183 ts_tablet_manager.cc:1905] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:54.354669 4180 tablet_replica.cc:330] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: stopping tablet replica
I20260812 02:34:54.355356 4180 raft_consensus.cc:2229] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 1 FOLLOWER]: Raft consensus shutting down.
Aug 12 02:34:54 dist-test-slave-vq2q krb5kdc[3521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.193@KRBTEST.COM
I20260812 02:34:54.356041 4180 raft_consensus.cc:2258] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:54.358891 4180 ts_tablet_manager.cc:1905] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:54.359514 4181 ts_tablet_manager.cc:1918] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:54.359793 4181 log.cc:1196] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/dd67a645b78141899f72baa056edd4ce
I20260812 02:34:54.360514 4181 ts_tablet_manager.cc:1939] T dd67a645b78141899f72baa056edd4ce P 4acf66e9d707492a87f777bee09de004: Deleting consensus metadata
I20260812 02:34:54.363315 3557 catalog_manager.cc:4841] TS 4acf66e9d707492a87f777bee09de004 (127.12.8.193:42851): tablet dd67a645b78141899f72baa056edd4ce (table test-table [id=e50c52afeb7b4f9fadeac0340141901b]) successfully deleted
I20260812 02:34:54.365170 4183 ts_tablet_manager.cc:1918] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:54.365284 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.332960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.12.8.193:42851)
0812 02:34:54.333225 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:54.333232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:54.342399 (+ 9167us) server_negotiation.cc:407] Connection header received
0812 02:34:54.342674 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:54.342681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:54.342837 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:54.342998 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:54.346736 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.348083 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.350465 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.351239 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.351340 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:54.356821 (+ 5481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:54.356844 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:54.356866 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:54.356917 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:54.361405 (+ 4488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:54.362399 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:54.362409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:54.362413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:54.362594 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:54.363102 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:54.363108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:54.363111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:54.363629 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:54.363957 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:54.364586 (+ 629us) server_negotiation.cc:299] Negotiation successful
0812 02:34:54.364842 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:34:54.365476 4183 log.cc:1196] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/dd67a645b78141899f72baa056edd4ce
I20260812 02:34:54.366219 4183 ts_tablet_manager.cc:1939] T dd67a645b78141899f72baa056edd4ce P 479d1314317d4b4e987303c26ab199db: Deleting consensus metadata
I20260812 02:34:54.366803 3801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:54.367657 4180 ts_tablet_manager.cc:1918] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:54.367923 4180 log.cc:1196] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/dd67a645b78141899f72baa056edd4ce
I20260812 02:34:54.368605 4180 ts_tablet_manager.cc:1939] T dd67a645b78141899f72baa056edd4ce P 9dad9550591c434ab9d6c937f3d768e1: Deleting consensus metadata
I20260812 02:34:54.368837 3557 catalog_manager.cc:4841] TS 479d1314317d4b4e987303c26ab199db (127.12.8.195:41989): tablet dd67a645b78141899f72baa056edd4ce (table test-table [id=e50c52afeb7b4f9fadeac0340141901b]) successfully deleted
I20260812 02:34:54.371527 3557 catalog_manager.cc:4841] TS 9dad9550591c434ab9d6c937f3d768e1 (127.12.8.194:42723): tablet dd67a645b78141899f72baa056edd4ce (table test-table [id=e50c52afeb7b4f9fadeac0340141901b]) successfully deleted
W20260812 02:34:54.373479 3557 catalog_manager.cc:4808] TS 9dad9550591c434ab9d6c937f3d768e1 (127.12.8.194:42723): delete failed for tablet dd67a645b78141899f72baa056edd4ce because the tablet was not found. No further retry: Not found: Tablet not found: dd67a645b78141899f72baa056edd4ce
I20260812 02:34:54.505261 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 3650
I20260812 02:34:54.538523 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 3816
Aug 12 02:34:54 dist-test-slave-vq2q krb5kdc[3521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.194@KRBTEST.COM
I20260812 02:34:54.558723 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.539679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.12.8.194:42723)
0812 02:34:54.539893 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:54.539898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:54.539947 (+ 49us) server_negotiation.cc:407] Connection header received
0812 02:34:54.540238 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:54.540243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:54.540364 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:54.540475 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:54.542479 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.543267 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.545358 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.545779 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.545908 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:54.552057 (+ 6149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:54.552077 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:54.552094 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:54.552151 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:54.555159 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:54.556308 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:54.556317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:54.556319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:54.556493 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:54.556945 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:54.556948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:54.556950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:54.557464 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:54.557718 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:54.558196 (+ 478us) server_negotiation.cc:299] Negotiation successful
0812 02:34:54.558411 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 02:34:54.559428 3961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:54.668782 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 3816
I20260812 02:34:54.695710 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 3976
Aug 12 02:34:54 dist-test-slave-vq2q krb5kdc[3521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.195@KRBTEST.COM
I20260812 02:34:54.720074 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.696987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.12.8.195:41989)
0812 02:34:54.697314 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:54.697321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:54.697403 (+ 82us) server_negotiation.cc:407] Connection header received
0812 02:34:54.697488 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:54.697494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:54.697598 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:54.697730 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:54.699422 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.700671 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.702715 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:54.703397 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:54.703600 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:54.710607 (+ 7007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:54.710633 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:54.710656 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:54.710731 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:54.715005 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:54.716441 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:54.716452 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:54.716455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:54.716883 (+ 428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:54.717606 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:54.717620 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:54.717622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:54.718325 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:54.718699 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:54.719292 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 02:34:54.719567 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 02:34:54.721465 4125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:54.870046 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 3976
I20260812 02:34:54.900089 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 3536
I20260812 02:34:54.901541 3609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:55.016002 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 3536
2026-08-12T02:34:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5011 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T02:34:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:55Z Disabled control of system clock
I20260812 02:34:55.137442 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46467
--webserver_interface=127.12.8.254
--webserver_port=0
--builtin_ntp_servers=127.12.8.212:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:34:55.600894 4195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:55.601348 4195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:55.601456 4195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:55.609782 4195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:55.609939 4195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:55.609982 4195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:55.610049 4195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:55.610100 4195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:55.620772 4195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:46467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:55.622620 4195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:55.625075 4195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:55.641005 4202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:55.641048 4205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:55.641005 4203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:55.642294 4195 server_base.cc:1034] running on GCE node
I20260812 02:34:55.643543 4195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:55.646170 4195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:55.647535 4195 hybrid_clock.cc:648] HybridClock initialized: now 1786502095647426 us; error 62 us; skew 500 ppm
I20260812 02:34:55.648028 4195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:55.651252 4195 webserver.cc:460] Webserver started at http://127.12.8.254:40309/ using document root <none> and password file <none>
I20260812 02:34:55.652215 4195 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:55.652344 4195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:55.652840 4195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:55.657135 4195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "76fe1223b7c043888aa99ea0e47d0fed"
format_stamp: "Formatted at 2026-08-12 02:34:55 on dist-test-slave-vq2q"
server_key: "2e0fc3b80d56bb8dbd4623cdf3a079ab"
server_key_iv: "89f242714042239a3822e4550e9f0ac0"
server_key_version: "encryptionkey@0"
I20260812 02:34:55.658221 4195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "76fe1223b7c043888aa99ea0e47d0fed"
format_stamp: "Formatted at 2026-08-12 02:34:55 on dist-test-slave-vq2q"
server_key: "2e0fc3b80d56bb8dbd4623cdf3a079ab"
server_key_iv: "89f242714042239a3822e4550e9f0ac0"
server_key_version: "encryptionkey@0"
I20260812 02:34:55.664160 4195 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 02:34:55.668028 4211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:55.669524 4195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:34:55.669683 4195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "76fe1223b7c043888aa99ea0e47d0fed"
format_stamp: "Formatted at 2026-08-12 02:34:55 on dist-test-slave-vq2q"
server_key: "2e0fc3b80d56bb8dbd4623cdf3a079ab"
server_key_iv: "89f242714042239a3822e4550e9f0ac0"
server_key_version: "encryptionkey@0"
I20260812 02:34:55.669878 4195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:55.695815 4195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:55.696826 4195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:55.697088 4195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:55.716159 4195 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:46467
I20260812 02:34:55.716167 4272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:46467 every 8 connection(s)
I20260812 02:34:55.717661 4195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:55.721570 4275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:55.727538 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 4195
I20260812 02:34:55.727732 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:34:55.728223 12323 external_mini_cluster.cc:1428] Setting key 0425e992277c91a7976c09e7d98a5381
I20260812 02:34:55.732903 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed: Bootstrap starting.
I20260812 02:34:55.736371 4275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:55.737751 4275 log.cc:824] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:55.741091 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed: No bootstrap required, opened a new log
I20260812 02:34:55.742180 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:55.730205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.12.8.254:46467)
0812 02:34:55.731277 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:55.731289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:55.731332 (+ 43us) server_negotiation.cc:407] Connection header received
0812 02:34:55.733044 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:55.733074 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:55.733575 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:55.733885 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:55.734536 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.735841 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.737722 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.738690 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.738890 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:55.739891 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:55.739898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:55.739912 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:55.740489 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:55.740563 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:55.740584 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:55.740670 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:55.740892 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 02:34:55.741219 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":377,"threads_started":1}
I20260812 02:34:55.748520 4275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fe1223b7c043888aa99ea0e47d0fed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46467 } }
I20260812 02:34:55.748869 4275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:55.748960 4275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fe1223b7c043888aa99ea0e47d0fed, State: Initialized, Role: FOLLOWER
I20260812 02:34:55.749584 4275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fe1223b7c043888aa99ea0e47d0fed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46467 } }
I20260812 02:34:55.749881 4275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:55.750020 4275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:55.750252 4275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:55.752398 4275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fe1223b7c043888aa99ea0e47d0fed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46467 } }
I20260812 02:34:55.752895 4275 leader_election.cc:302] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76fe1223b7c043888aa99ea0e47d0fed; no voters:
I20260812 02:34:55.753513 4275 leader_election.cc:288] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:55.753669 4281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:55.754364 4281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [term 1 LEADER]: Becoming Leader. State: Replica: 76fe1223b7c043888aa99ea0e47d0fed, State: Running, Role: LEADER
I20260812 02:34:55.754966 4281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fe1223b7c043888aa99ea0e47d0fed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46467 } }
I20260812 02:34:55.755654 4275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:55.758087 4283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76fe1223b7c043888aa99ea0e47d0fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fe1223b7c043888aa99ea0e47d0fed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46467 } } }
I20260812 02:34:55.758775 4283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:55.758162 4284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76fe1223b7c043888aa99ea0e47d0fed. Latest consensus state: current_term: 1 leader_uuid: "76fe1223b7c043888aa99ea0e47d0fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fe1223b7c043888aa99ea0e47d0fed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46467 } } }
I20260812 02:34:55.761770 4284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:55.762223 4296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:55.766901 4296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:55.781574 4296 catalog_manager.cc:1347] Generated new cluster ID: 8bfd114ad9724d5c81c4d2ba34ec0c2b
I20260812 02:34:55.781697 4296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:55.795713 4296 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:55.797348 4296 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:55.818091 4296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed: Generated new TSK 0
I20260812 02:34:55.819077 4296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:34:55.824589 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.193:0
--local_ip_for_outbound_sockets=127.12.8.193
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46467
--builtin_ntp_servers=127.12.8.212:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:34:56.202797 4308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:56.203104 4308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:56.203212 4308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:56.210757 4308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:56.210865 4308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:56.211009 4308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.193
I20260812 02:34:56.220300 4308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.193
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.193
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:56.221715 4308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:56.223562 4308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:56.236691 4313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:56.236668 4315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:56.237004 4318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:56.237498 4308 server_base.cc:1034] running on GCE node
I20260812 02:34:56.238129 4308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:56.239126 4308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:56.240356 4308 hybrid_clock.cc:648] HybridClock initialized: now 1786502096240307 us; error 34 us; skew 500 ppm
I20260812 02:34:56.240686 4308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:56.243170 4308 webserver.cc:460] Webserver started at http://127.12.8.193:33873/ using document root <none> and password file <none>
I20260812 02:34:56.243935 4308 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:56.244045 4308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:56.244367 4308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:56.248026 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/data/instance:
uuid: "97472edf2eca4466ae97f3961092938a"
format_stamp: "Formatted at 2026-08-12 02:34:56 on dist-test-slave-vq2q"
server_key: "0d5eb2e585365daedc1850100d36d7a1"
server_key_iv: "3d52d69b0de7278192209f0132cdb6d5"
server_key_version: "encryptionkey@0"
I20260812 02:34:56.248842 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance:
uuid: "97472edf2eca4466ae97f3961092938a"
format_stamp: "Formatted at 2026-08-12 02:34:56 on dist-test-slave-vq2q"
server_key: "0d5eb2e585365daedc1850100d36d7a1"
server_key_iv: "3d52d69b0de7278192209f0132cdb6d5"
server_key_version: "encryptionkey@0"
I20260812 02:34:56.254140 4308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 02:34:56.257853 4324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:56.259224 4308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:34:56.259383 4308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/wal
uuid: "97472edf2eca4466ae97f3961092938a"
format_stamp: "Formatted at 2026-08-12 02:34:56 on dist-test-slave-vq2q"
server_key: "0d5eb2e585365daedc1850100d36d7a1"
server_key_iv: "3d52d69b0de7278192209f0132cdb6d5"
server_key_version: "encryptionkey@0"
I20260812 02:34:56.259591 4308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:56.275794 4308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:56.276588 4308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:56.276885 4308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:56.277920 4308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:56.279387 4308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:56.279485 4308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:56.279592 4308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:56.279661 4308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:56.312243 4308 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.193:39623
I20260812 02:34:56.312260 4457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.193:39623 every 8 connection(s)
I20260812 02:34:56.313671 4308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:56.322785 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 4308
I20260812 02:34:56.322993 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/wal/instance
I20260812 02:34:56.323590 12323 external_mini_cluster.cc:1428] Setting key 277498cfaf1c7784f6327a3a271cfd8b
I20260812 02:34:56.327180 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.194:0
--local_ip_for_outbound_sockets=127.12.8.194
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46467
--builtin_ntp_servers=127.12.8.212:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:34:56.328877 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:56.316324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.193:53453 (local address 127.12.8.254:46467)
0812 02:34:56.316765 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:56.316770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:56.317952 (+ 1182us) server_negotiation.cc:407] Connection header received
0812 02:34:56.320511 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:56.320522 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:56.320637 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:56.320767 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:56.322236 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:56.323382 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:56.325624 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:56.326149 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:56.326250 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:56.327920 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:56.327925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:56.327928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:56.327948 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:56.327998 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:56.328015 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:56.328097 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:56.328418 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 02:34:56.328525 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":213,"threads_started":1}
I20260812 02:34:56.330017 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:56.316877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46467 (local address 127.12.8.193:53453)
0812 02:34:56.317792 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:56.317836 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:56.319880 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:56.321070 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:56.321083 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:56.321229 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:56.322051 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:56.322077 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:56.323579 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:56.323585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:56.325495 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:56.325510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:56.326506 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:56.326511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:56.326780 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:56.327561 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:56.327564 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:56.327609 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:56.327697 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:56.328148 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:56.328160 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:56.328169 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:34:56.328340 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 02:34:56.328674 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":290,"threads_started":1}
I20260812 02:34:56.331965 4458 heartbeater.cc:347] Connected to a master server at 127.12.8.254:46467
I20260812 02:34:56.332299 4458 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:56.333117 4458 heartbeater.cc:511] Master 127.12.8.254:46467 requested a full tablet report, sending...
I20260812 02:34:56.335474 4233 ts_manager.cc:194] Registered new tserver with Master: 97472edf2eca4466ae97f3961092938a (127.12.8.193:39623)
I20260812 02:34:56.337393 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.193:53453
W20260812 02:34:56.676376 4463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:56.676640 4463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:56.676749 4463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:56.684058 4463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:56.684175 4463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:56.684300 4463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.194
I20260812 02:34:56.693472 4463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.194
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.194
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:56.694813 4463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:56.696592 4463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:56.710629 4468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:56.710847 4469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:56.711463 4471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:56.713260 4463 server_base.cc:1034] running on GCE node
I20260812 02:34:56.713928 4463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:56.714957 4463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:56.716315 4463 hybrid_clock.cc:648] HybridClock initialized: now 1786502096716219 us; error 79 us; skew 500 ppm
I20260812 02:34:56.716681 4463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:56.719311 4463 webserver.cc:460] Webserver started at http://127.12.8.194:39017/ using document root <none> and password file <none>
I20260812 02:34:56.720032 4463 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:56.720130 4463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:56.720463 4463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:56.723595 4463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/data/instance:
uuid: "b43b7461f0114e3aa3a1ec386af28acb"
format_stamp: "Formatted at 2026-08-12 02:34:56 on dist-test-slave-vq2q"
server_key: "b754c616552c7407cef73f9570b3720f"
server_key_iv: "51cd2b4ad8c120cbc1c56e4960153eb6"
server_key_version: "encryptionkey@0"
I20260812 02:34:56.724426 4463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance:
uuid: "b43b7461f0114e3aa3a1ec386af28acb"
format_stamp: "Formatted at 2026-08-12 02:34:56 on dist-test-slave-vq2q"
server_key: "b754c616552c7407cef73f9570b3720f"
server_key_iv: "51cd2b4ad8c120cbc1c56e4960153eb6"
server_key_version: "encryptionkey@0"
I20260812 02:34:56.730075 4463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260812 02:34:56.733922 4478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:56.735510 4463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 02:34:56.735695 4463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/wal
uuid: "b43b7461f0114e3aa3a1ec386af28acb"
format_stamp: "Formatted at 2026-08-12 02:34:56 on dist-test-slave-vq2q"
server_key: "b754c616552c7407cef73f9570b3720f"
server_key_iv: "51cd2b4ad8c120cbc1c56e4960153eb6"
server_key_version: "encryptionkey@0"
I20260812 02:34:56.735940 4463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:56.748749 4463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:56.749653 4463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:56.749945 4463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:56.751003 4463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:56.752580 4463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:56.752679 4463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:56.752790 4463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:56.752864 4463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:56.789422 4463 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.194:44011
I20260812 02:34:56.789515 4607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.194:44011 every 8 connection(s)
I20260812 02:34:56.790791 4463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:56.793262 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 4463
I20260812 02:34:56.793469 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/wal/instance
I20260812 02:34:56.794064 12323 external_mini_cluster.cc:1428] Setting key 9d7eec3c7f065e2de4dd15bf5a995825
I20260812 02:34:56.798183 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.195:0
--local_ip_for_outbound_sockets=127.12.8.195
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46467
--builtin_ntp_servers=127.12.8.212:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:34:56.808297 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:56.794151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:57151 (local address 127.12.8.254:46467)
0812 02:34:56.794394 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:56.794400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:56.796728 (+ 2328us) server_negotiation.cc:407] Connection header received
0812 02:34:56.799623 (+ 2895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:56.799628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:56.799728 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:56.799848 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:56.801701 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:56.802497 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:56.804928 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:56.805403 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:56.805515 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:56.807478 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:56.807484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:56.807487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:56.807509 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:56.807545 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:56.807557 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:56.807639 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:56.807938 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 02:34:56.808030 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 02:34:56.810248 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:56.794674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46467 (local address 127.12.8.194:57151)
0812 02:34:56.796511 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:56.796568 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:56.799189 (+ 2621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:56.800207 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:56.800224 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:56.800386 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:56.801458 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:56.801488 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:56.802696 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:56.802703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:56.804723 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:56.804744 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:56.805680 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:56.805688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:56.806036 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:56.806964 (+ 928us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:56.806968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:56.807039 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:56.807192 (+ 153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:56.807668 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:56.807680 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:56.807689 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:34:56.807926 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 02:34:56.808500 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":800,"threads_started":1}
I20260812 02:34:56.812108 4608 heartbeater.cc:347] Connected to a master server at 127.12.8.254:46467
I20260812 02:34:56.812463 4608 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:56.813331 4608 heartbeater.cc:511] Master 127.12.8.254:46467 requested a full tablet report, sending...
I20260812 02:34:56.815059 4233 ts_manager.cc:194] Registered new tserver with Master: b43b7461f0114e3aa3a1ec386af28acb (127.12.8.194:44011)
I20260812 02:34:56.816337 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.194:57151
W20260812 02:34:57.155210 4612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:57.155508 4612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:57.155602 4612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:57.163039 4612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:57.163153 4612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:57.163278 4612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.195
I20260812 02:34:57.172680 4612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.195
--webserver_port=0
--tserver_master_addrs=127.12.8.254:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.195
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:34:57.174106 4612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:57.175945 4612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:57.190968 4617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:57.190828 4621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:57.190891 4619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:57.192799 4612 server_base.cc:1034] running on GCE node
I20260812 02:34:57.193581 4612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:57.194766 4612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:57.196019 4612 hybrid_clock.cc:648] HybridClock initialized: now 1786502097195967 us; error 53 us; skew 500 ppm
I20260812 02:34:57.196388 4612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 02:34:57.198936 4612 webserver.cc:460] Webserver started at http://127.12.8.195:39691/ using document root <none> and password file <none>
I20260812 02:34:57.199666 4612 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:57.199772 4612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:57.200104 4612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:57.203146 4612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/data/instance:
uuid: "f4762f15ed18421782b6f9a5bec43c21"
format_stamp: "Formatted at 2026-08-12 02:34:57 on dist-test-slave-vq2q"
server_key: "88ab72f1cdbc71250f9e1830738c858d"
server_key_iv: "6217c4030fed5922b7c26ad303b271f4"
server_key_version: "encryptionkey@0"
I20260812 02:34:57.204231 4612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance:
uuid: "f4762f15ed18421782b6f9a5bec43c21"
format_stamp: "Formatted at 2026-08-12 02:34:57 on dist-test-slave-vq2q"
server_key: "88ab72f1cdbc71250f9e1830738c858d"
server_key_iv: "6217c4030fed5922b7c26ad303b271f4"
server_key_version: "encryptionkey@0"
I20260812 02:34:57.210283 4612 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 02:34:57.214334 4629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:57.216001 4612 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:34:57.216193 4612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/wal
uuid: "f4762f15ed18421782b6f9a5bec43c21"
format_stamp: "Formatted at 2026-08-12 02:34:57 on dist-test-slave-vq2q"
server_key: "88ab72f1cdbc71250f9e1830738c858d"
server_key_iv: "6217c4030fed5922b7c26ad303b271f4"
server_key_version: "encryptionkey@0"
I20260812 02:34:57.216416 4612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:57.237051 4612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:57.237944 4612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:57.238243 4612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:57.239375 4612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:57.240969 4612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:57.241067 4612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:57.241214 4612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:57.241297 4612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 02:34:57.277628 4612 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.195:33989
I20260812 02:34:57.277709 4766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.195:33989 every 8 connection(s)
I20260812 02:34:57.278964 4612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:57.283852 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 4612
I20260812 02:34:57.284094 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/wal/instance
I20260812 02:34:57.284767 12323 external_mini_cluster.cc:1428] Setting key a28158dbe7965b0f25b4321a59a6afa7
I20260812 02:34:57.292670 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.282275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.195:32861 (local address 127.12.8.254:46467)
0812 02:34:57.282522 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.282528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.284373 (+ 1845us) server_negotiation.cc:407] Connection header received
0812 02:34:57.286372 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.286378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.286468 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.286556 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:57.287653 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.288427 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.290017 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.290411 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.290518 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.291812 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:57.291817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:57.291820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:57.291838 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:57.291870 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:57.291881 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:57.291955 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.292171 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.292282 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 02:34:57.293640 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.282957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:46467 (local address 127.12.8.195:32861)
0812 02:34:57.284186 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.284252 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:57.286085 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:57.286808 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:57.286816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:57.286903 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:57.287480 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.287495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.288584 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.288588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.289877 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.289890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.290635 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.290644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.290857 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.291498 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:34:57.291501 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:57.291541 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:34:57.291622 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:57.291972 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:57.291984 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:57.291991 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:34:57.292145 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 02:34:57.292525 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":404,"threads_started":1}
I20260812 02:34:57.295470 4767 heartbeater.cc:347] Connected to a master server at 127.12.8.254:46467
I20260812 02:34:57.295855 4767 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:57.296798 4767 heartbeater.cc:511] Master 127.12.8.254:46467 requested a full tablet report, sending...
I20260812 02:34:57.298581 4233 ts_manager.cc:194] Registered new tserver with Master: f4762f15ed18421782b6f9a5bec43c21 (127.12.8.195:33989)
I20260812 02:34:57.299715 4233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.195:32861
I20260812 02:34:57.301481 12323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:57.313364 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.307298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.12.8.254:46467)
0812 02:34:57.307539 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.307547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.308033 (+ 486us) server_negotiation.cc:407] Connection header received
0812 02:34:57.308221 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.308225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.308325 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.308426 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:57.309030 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.309804 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.311102 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.311522 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.311630 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.312669 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:57.312675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:57.312679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:57.312697 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:57.312729 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:57.312742 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:57.312814 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.312947 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.313036 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 02:34:57.318724 4233 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:57.327225 4233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:57.340615 4458 heartbeater.cc:503] Master 127.12.8.254:46467 was elected leader, sending a full tablet report...
I20260812 02:34:57.356092 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.344195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.12.8.193:39623)
0812 02:34:57.345227 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.345237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.347098 (+ 1861us) server_negotiation.cc:407] Connection header received
0812 02:34:57.347221 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.347229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.347514 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.347707 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:57.348503 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.350107 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.352817 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.354395 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.354503 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.354794 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.355507 (+ 713us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.355692 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":1049,"threads_started":1}
I20260812 02:34:57.356086 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.343811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39623 (local address 127.0.0.1:41738)
0812 02:34:57.345300 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.345357 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:57.346696 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:57.347748 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:57.347755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:57.347788 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:57.348290 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.348305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.350277 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.350283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.352587 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.352600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.354999 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.355005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.355207 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.355263 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 02:34:57.355446 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 02:34:57.355658 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":36,"thread_start_us":661,"threads_started":1}
I20260812 02:34:57.357533 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.345563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:33989 (local address 127.0.0.1:34356)
0812 02:34:57.346612 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.346632 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:57.346780 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:57.349617 (+ 2837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:57.349622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:57.349650 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:57.349952 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.349963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.351837 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.351843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.353980 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.353997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.356696 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.356702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.356888 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.356921 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 02:34:57.356997 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 02:34:57.357140 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":52,"thread_start_us":3068,"threads_started":1}
I20260812 02:34:57.357823 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.346115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.12.8.195:33989)
0812 02:34:57.348675 (+ 2560us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.348684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.348720 (+ 36us) server_negotiation.cc:407] Connection header received
0812 02:34:57.348819 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.348838 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.349070 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.349334 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:57.350162 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.351675 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.354286 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.356447 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.356634 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.356992 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.357082 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.357370 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2362,"thread_start_us":2168,"threads_started":1}
I20260812 02:34:57.361974 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.345065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.12.8.194:44011)
0812 02:34:57.345907 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.345913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.349139 (+ 3226us) server_negotiation.cc:407] Connection header received
0812 02:34:57.349338 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.349347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.349634 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.349857 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:57.351152 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.352591 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.355324 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.356810 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.356958 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.357221 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.357877 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.361588 (+ 3711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":536,"threads_started":1}
I20260812 02:34:57.361511 4376 tablet_service.cc:1467] Processing CreateTablet for tablet d6c6a9c32b084195ab6eeaf33da41a4a (DEFAULT_TABLE table=test-table [id=e59bcc54c8974b08b2406a2f107be068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:57.361711 4688 tablet_service.cc:1467] Processing CreateTablet for tablet d6c6a9c32b084195ab6eeaf33da41a4a (DEFAULT_TABLE table=test-table [id=e59bcc54c8974b08b2406a2f107be068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:57.362880 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.344823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.194:44011 (local address 127.0.0.1:41690)
0812 02:34:57.348966 (+ 4143us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.348986 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:57.349144 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:57.350190 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:57.350194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:57.350221 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:57.350875 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.350890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.352775 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.352781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.355005 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.355018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.357383 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.357390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.357636 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.357679 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 02:34:57.362278 (+ 4599us) client_negotiation.cc:240] Negotiation successful
0812 02:34:57.362457 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4009,"mutex_wait_us":1156,"thread_start_us":358,"threads_started":1}
I20260812 02:34:57.363544 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c6a9c32b084195ab6eeaf33da41a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:57.363602 4376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c6a9c32b084195ab6eeaf33da41a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:57.367218 4533 tablet_service.cc:1467] Processing CreateTablet for tablet d6c6a9c32b084195ab6eeaf33da41a4a (DEFAULT_TABLE table=test-table [id=e59bcc54c8974b08b2406a2f107be068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:57.368722 4533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c6a9c32b084195ab6eeaf33da41a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:57.375752 4789 tablet_bootstrap.cc:492] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Bootstrap starting.
I20260812 02:34:57.379467 4789 tablet_bootstrap.cc:654] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:57.380560 4789 log.cc:824] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:57.383685 4789 tablet_bootstrap.cc:492] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: No bootstrap required, opened a new log
I20260812 02:34:57.383998 4789 ts_tablet_manager.cc:1397] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260812 02:34:57.386153 4791 tablet_bootstrap.cc:492] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Bootstrap starting.
I20260812 02:34:57.390748 4791 tablet_bootstrap.cc:654] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:57.391937 4793 tablet_bootstrap.cc:492] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Bootstrap starting.
I20260812 02:34:57.392081 4791 log.cc:824] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:57.393949 4789 raft_consensus.cc:348] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.394336 4789 raft_consensus.cc:374] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:57.394428 4789 raft_consensus.cc:729] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4762f15ed18421782b6f9a5bec43c21, State: Initialized, Role: FOLLOWER
I20260812 02:34:57.394992 4789 consensus_queue.cc:260] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.396051 4767 heartbeater.cc:503] Master 127.12.8.254:46467 was elected leader, sending a full tablet report...
I20260812 02:34:57.396579 4793 tablet_bootstrap.cc:654] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:57.396724 4789 ts_tablet_manager.cc:1428] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260812 02:34:57.396744 4791 tablet_bootstrap.cc:492] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: No bootstrap required, opened a new log
I20260812 02:34:57.397109 4791 ts_tablet_manager.cc:1397] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260812 02:34:57.398178 4793 log.cc:824] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:57.404857 4793 tablet_bootstrap.cc:492] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: No bootstrap required, opened a new log
I20260812 02:34:57.405292 4793 ts_tablet_manager.cc:1397] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260812 02:34:57.408444 4791 raft_consensus.cc:348] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.409005 4791 raft_consensus.cc:374] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:57.409116 4791 raft_consensus.cc:729] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43b7461f0114e3aa3a1ec386af28acb, State: Initialized, Role: FOLLOWER
I20260812 02:34:57.409942 4791 consensus_queue.cc:260] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.411228 4608 heartbeater.cc:503] Master 127.12.8.254:46467 was elected leader, sending a full tablet report...
I20260812 02:34:57.412228 4791 ts_tablet_manager.cc:1428] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260812 02:34:57.415656 4793 raft_consensus.cc:348] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.416155 4793 raft_consensus.cc:374] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:57.416270 4793 raft_consensus.cc:729] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97472edf2eca4466ae97f3961092938a, State: Initialized, Role: FOLLOWER
I20260812 02:34:57.416962 4793 consensus_queue.cc:260] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.418820 4793 ts_tablet_manager.cc:1428] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260812 02:34:57.530855 4768 tablet.cc:2369] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:34:57.543134 4609 tablet.cc:2369] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:34:57.566272 4459 tablet.cc:2369] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:57.592037 4797 raft_consensus.cc:482] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:57.592442 4797 raft_consensus.cc:504] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.594625 4797 leader_election.cc:288] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97472edf2eca4466ae97f3961092938a (127.12.8.193:39623), f4762f15ed18421782b6f9a5bec43c21 (127.12.8.195:33989)
I20260812 02:34:57.603850 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.595168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:37125 (local address 127.12.8.193:39623)
0812 02:34:57.595366 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.595371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.597064 (+ 1693us) server_negotiation.cc:407] Connection header received
0812 02:34:57.597359 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.597364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.597446 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.597531 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:57.598158 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.599073 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.601719 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.602617 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.602791 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.602876 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.603396 (+ 520us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.603544 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:34:57.604794 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.595239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.193:39623 (local address 127.12.8.194:37125)
0812 02:34:57.596898 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.596917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:57.597095 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:57.597566 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:57.597571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:57.597598 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:57.598021 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.598032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.599220 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.599232 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.601520 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.601537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.602827 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.602835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.603160 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.603201 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 02:34:57.604221 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0812 02:34:57.604391 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":1,"thread_start_us":1336,"threads_started":1}
I20260812 02:34:57.605747 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.597494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.194:33287 (local address 127.12.8.195:33989)
0812 02:34:57.597696 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.597702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.598300 (+ 598us) server_negotiation.cc:407] Connection header received
0812 02:34:57.598640 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.598647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.598727 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.598782 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:57.599806 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.600892 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.603305 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.604436 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.604624 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.604709 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.605249 (+ 540us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.605432 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:34:57.606417 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.597135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.195:33989 (local address 127.12.8.194:33287)
0812 02:34:57.598182 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.598203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:57.598433 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:57.598951 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:57.598954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:57.598975 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:57.599517 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.599532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.601082 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.601088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.603054 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:57.603072 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.604682 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.604689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:57.604910 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.605012 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 02:34:57.605988 (+ 976us) client_negotiation.cc:240] Negotiation successful
0812 02:34:57.606132 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":49,"thread_start_us":1258,"threads_started":1}
I20260812 02:34:57.606483 4400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c6a9c32b084195ab6eeaf33da41a4a" candidate_uuid: "b43b7461f0114e3aa3a1ec386af28acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97472edf2eca4466ae97f3961092938a" is_pre_election: true
I20260812 02:34:57.607126 4400 raft_consensus.cc:2454] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43b7461f0114e3aa3a1ec386af28acb in term 0.
I20260812 02:34:57.607959 4481 leader_election.cc:302] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97472edf2eca4466ae97f3961092938a, b43b7461f0114e3aa3a1ec386af28acb; no voters:
I20260812 02:34:57.608122 4712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c6a9c32b084195ab6eeaf33da41a4a" candidate_uuid: "b43b7461f0114e3aa3a1ec386af28acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4762f15ed18421782b6f9a5bec43c21" is_pre_election: true
I20260812 02:34:57.608558 4797 raft_consensus.cc:2781] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:57.608705 4797 raft_consensus.cc:482] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:57.608780 4797 raft_consensus.cc:3037] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:57.608783 4712 raft_consensus.cc:2454] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43b7461f0114e3aa3a1ec386af28acb in term 0.
I20260812 02:34:57.612429 4797 raft_consensus.cc:504] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.613293 4797 leader_election.cc:288] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [CANDIDATE]: Term 1 election: Requested vote from peers 97472edf2eca4466ae97f3961092938a (127.12.8.193:39623), f4762f15ed18421782b6f9a5bec43c21 (127.12.8.195:33989)
I20260812 02:34:57.613885 4400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c6a9c32b084195ab6eeaf33da41a4a" candidate_uuid: "b43b7461f0114e3aa3a1ec386af28acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97472edf2eca4466ae97f3961092938a"
I20260812 02:34:57.614125 4400 raft_consensus.cc:3037] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:57.614518 4712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c6a9c32b084195ab6eeaf33da41a4a" candidate_uuid: "b43b7461f0114e3aa3a1ec386af28acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4762f15ed18421782b6f9a5bec43c21"
I20260812 02:34:57.614799 4712 raft_consensus.cc:3037] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:57.617857 4712 raft_consensus.cc:2454] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43b7461f0114e3aa3a1ec386af28acb in term 1.
I20260812 02:34:57.618009 4400 raft_consensus.cc:2454] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43b7461f0114e3aa3a1ec386af28acb in term 1.
I20260812 02:34:57.618618 4481 leader_election.cc:302] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b43b7461f0114e3aa3a1ec386af28acb, f4762f15ed18421782b6f9a5bec43c21; no voters:
I20260812 02:34:57.619035 4797 raft_consensus.cc:2781] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:57.619642 4797 raft_consensus.cc:686] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 1 LEADER]: Becoming Leader. State: Replica: b43b7461f0114e3aa3a1ec386af28acb, State: Running, Role: LEADER
I20260812 02:34:57.620369 4797 consensus_queue.cc:237] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } }
I20260812 02:34:57.626698 4233 catalog_manager.cc:5495] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb reported cstate change: term changed from 0 to 1, leader changed from <none> to b43b7461f0114e3aa3a1ec386af28acb (127.12.8.194). New cstate: current_term: 1 leader_uuid: "b43b7461f0114e3aa3a1ec386af28acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43b7461f0114e3aa3a1ec386af28acb" member_type: VOTER last_known_addr { host: "127.12.8.194" port: 44011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 } health_report { overall_health: UNKNOWN } } }
I20260812 02:34:57.661471 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.655324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.12.8.194:44011)
0812 02:34:57.655562 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.655569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.655595 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:34:57.655884 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.655888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.656000 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.656147 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:57.656578 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.657383 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.659020 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.659442 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.659555 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.660134 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:57.660815 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:57.660952 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.661056 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.661144 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:34:57.672312 4400 raft_consensus.cc:1261] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 1 FOLLOWER]: Refusing update from remote peer b43b7461f0114e3aa3a1ec386af28acb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:57.672384 4712 raft_consensus.cc:1261] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 1 FOLLOWER]: Refusing update from remote peer b43b7461f0114e3aa3a1ec386af28acb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:57.673561 4801 consensus_queue.cc:1035] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "97472edf2eca4466ae97f3961092938a" member_type: VOTER last_known_addr { host: "127.12.8.193" port: 39623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:57.674039 4797 consensus_queue.cc:1035] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4762f15ed18421782b6f9a5bec43c21" member_type: VOTER last_known_addr { host: "127.12.8.195" port: 33989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:57.687408 4806 mvcc.cc:205] Tried to move back new op lower bound from 7317512592052862976 to 7317512591858966528. Current Snapshot: MvccSnapshot[applied={T|T < 7317512592052862976}]
I20260812 02:34:57.705133 4233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:57.705467 4233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:57.711329 4233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76fe1223b7c043888aa99ea0e47d0fed: Sending DeleteTablet for 3 replicas of tablet d6c6a9c32b084195ab6eeaf33da41a4a
I20260812 02:34:57.713109 4533 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c6a9c32b084195ab6eeaf33da41a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:57 UTC) from {username='slave'} at 127.0.0.1:41690
I20260812 02:34:57.713485 4376 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c6a9c32b084195ab6eeaf33da41a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:57 UTC) from {username='slave'} at 127.0.0.1:41738
I20260812 02:34:57.714279 4688 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c6a9c32b084195ab6eeaf33da41a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:57 UTC) from {username='slave'} at 127.0.0.1:34356
I20260812 02:34:57.714721 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 4308
I20260812 02:34:57.741734 4822 tablet_replica.cc:330] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: stopping tablet replica
I20260812 02:34:57.742336 4822 raft_consensus.cc:2229] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:57.742956 4822 raft_consensus.cc:2258] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:57.745306 4822 ts_tablet_manager.cc:1905] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:57.749907 4819 tablet_replica.cc:330] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: stopping tablet replica
I20260812 02:34:57.750403 4819 raft_consensus.cc:2229] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:34:57.751113 4819 raft_consensus.cc:2258] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:57.753937 4823 tablet_replica.cc:330] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: stopping tablet replica
I20260812 02:34:57.754400 4823 raft_consensus.cc:2229] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:57.754900 4823 raft_consensus.cc:2258] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:57.755545 4822 ts_tablet_manager.cc:1918] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:57.755775 4822 log.cc:1196] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-0/wal/wals/d6c6a9c32b084195ab6eeaf33da41a4a
I20260812 02:34:57.756347 4822 ts_tablet_manager.cc:1939] T d6c6a9c32b084195ab6eeaf33da41a4a P 97472edf2eca4466ae97f3961092938a: Deleting consensus metadata
I20260812 02:34:57.756373 4819 ts_tablet_manager.cc:1905] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:57.757318 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.719778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.12.8.193:39623)
0812 02:34:57.719979 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.719986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.736463 (+ 16477us) server_negotiation.cc:407] Connection header received
0812 02:34:57.736689 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.736695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.736852 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.736949 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:57.737722 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.738961 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.742647 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.743305 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.743403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.755449 (+ 12046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:57.755462 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:57.755480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:57.756307 (+ 827us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:57.756382 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:57.756412 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:57.756516 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.756793 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.756924 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:34:57.758468 4216 catalog_manager.cc:4841] TS 97472edf2eca4466ae97f3961092938a (127.12.8.193:39623): tablet d6c6a9c32b084195ab6eeaf33da41a4a (table test-table [id=e59bcc54c8974b08b2406a2f107be068]) successfully deleted
I20260812 02:34:57.758689 4450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:57.760447 4823 ts_tablet_manager.cc:1905] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:57.770120 4823 ts_tablet_manager.cc:1918] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:57.770402 4823 log.cc:1196] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-2/wal/wals/d6c6a9c32b084195ab6eeaf33da41a4a
I20260812 02:34:57.771062 4823 ts_tablet_manager.cc:1939] T d6c6a9c32b084195ab6eeaf33da41a4a P f4762f15ed18421782b6f9a5bec43c21: Deleting consensus metadata
I20260812 02:34:57.773398 4216 catalog_manager.cc:4841] TS f4762f15ed18421782b6f9a5bec43c21 (127.12.8.195:33989): tablet d6c6a9c32b084195ab6eeaf33da41a4a (table test-table [id=e59bcc54c8974b08b2406a2f107be068]) successfully deleted
I20260812 02:34:57.774688 4819 ts_tablet_manager.cc:1918] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:57.775594 4819 log.cc:1196] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Deleting WAL directory at /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501741404412-12323-0/minicluster-data/ts-1/wal/wals/d6c6a9c32b084195ab6eeaf33da41a4a
I20260812 02:34:57.776655 4819 ts_tablet_manager.cc:1939] T d6c6a9c32b084195ab6eeaf33da41a4a P b43b7461f0114e3aa3a1ec386af28acb: Deleting consensus metadata
I20260812 02:34:57.779574 4215 catalog_manager.cc:4841] TS b43b7461f0114e3aa3a1ec386af28acb (127.12.8.194:44011): tablet d6c6a9c32b084195ab6eeaf33da41a4a (table test-table [id=e59bcc54c8974b08b2406a2f107be068]) successfully deleted
I20260812 02:34:57.885586 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 4308
I20260812 02:34:57.921620 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 4463
I20260812 02:34:57.929399 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.922616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.12.8.194:44011)
0812 02:34:57.922873 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:57.922878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:57.922894 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:34:57.922957 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:57.922961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:57.923049 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:57.923140 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:57.923584 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.924421 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.925779 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:57.926226 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:57.926355 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:57.927839 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:57.927849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:57.927865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:57.928543 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:57.928597 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:57.928619 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:57.928731 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:57.928883 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 02:34:57.929003 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260812 02:34:57.930194 4603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:58.049248 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 4463
I20260812 02:34:58.081802 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 4612
I20260812 02:34:58.089425 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:58.082672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.12.8.195:33989)
0812 02:34:58.082921 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:58.082927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:58.082972 (+ 45us) server_negotiation.cc:407] Connection header received
0812 02:34:58.083160 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:58.083165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:58.083275 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:58.083386 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:58.083793 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:58.084586 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:58.086042 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:58.086512 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:58.086631 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:58.087765 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:58.087774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:58.087793 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:34:58.088519 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:58.088590 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:34:58.088612 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:58.088720 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:58.088890 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 02:34:58.089005 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 02:34:58.090322 4761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:58.190837 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 4612
I20260812 02:34:58.214375 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 4195
I20260812 02:34:58.215794 4268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:34:58.344419 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 4195
2026-08-12T02:34:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3336 ms)
[----------] 6 tests from EncryptionPolicyTest (25312 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:34:58 dist-test-slave-vq2q krb5kdc[4834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:34:58 dist-test-slave-vq2q krb5kdc[4834](info): set up 2 sockets
Aug 12 02:34:58 dist-test-slave-vq2q krb5kdc[4834](info): commencing operation
krb5kdc: starting...
W20260812 02:34:59.557243 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.001s sys 0.076s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:34:59 dist-test-slave-vq2q krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:34:59.987361 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:00.429845 4849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:00.430270 4849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:00.430380 4849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:00.439131 4849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:00.439329 4849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:00.439392 4849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:00.439460 4849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:00.439509 4849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:00.451203 4849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:00.452989 4849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:00.454686 4849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:00.468873 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:00.468943 4855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:00.468920 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:00.469843 4849 server_base.cc:1034] running on GCE node
I20260812 02:35:00.470867 4849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:00.472795 4849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:00.474082 4849 hybrid_clock.cc:648] HybridClock initialized: now 1786502100474008 us; error 62 us; skew 500 ppm
I20260812 02:35:00.474545 4849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:00 dist-test-slave-vq2q krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:00.482345 4849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:00.484408 4849 webserver.cc:460] Webserver started at http://127.0.0.1:46181/ using document root <none> and password file <none>
I20260812 02:35:00.485340 4849 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:00.485477 4849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:00.485814 4849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:00.489622 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "18695c3b3cb64214b475dba57d6bd162"
format_stamp: "Formatted at 2026-08-12 02:35:00 on dist-test-slave-vq2q"
server_key: "f8aa87709cb7a569d93fb8e91c17cc3f"
server_key_iv: "862f412c4981e03cb727cfeee5f9d283"
server_key_version: "encryptionkey@0"
I20260812 02:35:00.490518 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "18695c3b3cb64214b475dba57d6bd162"
format_stamp: "Formatted at 2026-08-12 02:35:00 on dist-test-slave-vq2q"
server_key: "f8aa87709cb7a569d93fb8e91c17cc3f"
server_key_iv: "862f412c4981e03cb727cfeee5f9d283"
server_key_version: "encryptionkey@0"
I20260812 02:35:00.497033 4849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 02:35:00.501389 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:00.503059 4849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:35:00.503283 4849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "18695c3b3cb64214b475dba57d6bd162"
format_stamp: "Formatted at 2026-08-12 02:35:00 on dist-test-slave-vq2q"
server_key: "f8aa87709cb7a569d93fb8e91c17cc3f"
server_key_iv: "862f412c4981e03cb727cfeee5f9d283"
server_key_version: "encryptionkey@0"
I20260812 02:35:00.503518 4849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:00.523262 4849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:00.528947 4849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:00.529304 4849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:00.550815 4849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39231
I20260812 02:35:00.550839 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39231 every 8 connection(s)
I20260812 02:35:00.552440 4849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:00.555946 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 4849
I20260812 02:35:00.556252 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:00.557104 12323 external_mini_cluster.cc:1428] Setting key d280ad5ab69d8f43f31592c3363de615
I20260812 02:35:00.557471 4932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:00.574226 4932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162: Bootstrap starting.
Aug 12 02:35:00 dist-test-slave-vq2q krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:35:00.579313 4932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:00.581089 4932 log.cc:824] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:00.585628 4932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162: No bootstrap required, opened a new log
I20260812 02:35:00.587345 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:00.559504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54789 (local address 127.0.0.1:39231)
0812 02:35:00.560477 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:00.560493 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:00.560551 (+ 58us) server_negotiation.cc:407] Connection header received
0812 02:35:00.562973 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:00.563003 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:00.563546 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:00.563983 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:00.566141 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:00.567623 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:00.569496 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:00.570365 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:00.570523 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:00.576009 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:00.576024 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:00.576050 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:00.576146 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:00.581021 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:00.582483 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:00.582493 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:00.582502 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:00.582726 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:00.583281 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:00.583285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:00.583288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:00.584245 (+ 957us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:00.584653 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:00.585221 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 02:35:00.585658 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":435,"threads_started":1}
I20260812 02:35:00.592399 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:00.596822 4932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18695c3b3cb64214b475dba57d6bd162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39231 } }
I20260812 02:35:00.597585 4932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:00.597715 4932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18695c3b3cb64214b475dba57d6bd162, State: Initialized, Role: FOLLOWER
I20260812 02:35:00.598623 4932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18695c3b3cb64214b475dba57d6bd162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39231 } }
I20260812 02:35:00.598949 4932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:00.599184 4932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:00.599408 4932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:00.603077 4932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18695c3b3cb64214b475dba57d6bd162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39231 } }
I20260812 02:35:00.603817 4932 leader_election.cc:302] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18695c3b3cb64214b475dba57d6bd162; no voters:
I20260812 02:35:00.604671 4932 leader_election.cc:288] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:00.604784 4942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:00.607632 4932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:00.611841 4942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [term 1 LEADER]: Becoming Leader. State: Replica: 18695c3b3cb64214b475dba57d6bd162, State: Running, Role: LEADER
I20260812 02:35:00.612782 4942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18695c3b3cb64214b475dba57d6bd162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39231 } }
I20260812 02:35:00.617511 4943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18695c3b3cb64214b475dba57d6bd162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18695c3b3cb64214b475dba57d6bd162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39231 } } }
I20260812 02:35:00.618234 4943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:00.617249 4944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18695c3b3cb64214b475dba57d6bd162. Latest consensus state: current_term: 1 leader_uuid: "18695c3b3cb64214b475dba57d6bd162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18695c3b3cb64214b475dba57d6bd162" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39231 } } }
I20260812 02:35:00.619014 4944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:00.620942 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:00.599303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36007 (local address 127.0.0.1:39231)
0812 02:35:00.599557 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:00.599565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:00.600158 (+ 593us) server_negotiation.cc:407] Connection header received
0812 02:35:00.600369 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:00.600374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:00.600454 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:00.600554 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:00.602506 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:00.603408 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:00.605300 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:00.605995 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:00.606110 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:00.610482 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:00.610491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:00.610498 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:00.610559 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:00.614939 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:00.616278 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:00.616286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:00.616290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:00.616492 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:00.617161 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:00.617167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:00.617171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:00.618138 (+ 967us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:00.619998 (+ 1860us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:00.620228 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 02:35:00.620489 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:35:00.630066 4957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:00.635198 4957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:00.645745 4957 catalog_manager.cc:1347] Generated new cluster ID: 73d834873d8b4649845e1a2ce4d9f9be
I20260812 02:35:00.645905 4957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:00.672220 4957 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:00.673949 4957 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:00.685966 4957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18695c3b3cb64214b475dba57d6bd162: Generated new TSK 0
I20260812 02:35:00.687382 4957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:00.701892 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 4849
I20260812 02:35:00.703469 4927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:00.805241 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 4849
W20260812 02:35:00.826834 4830 connection.cc:535] client connection to 127.0.0.1:39231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39231): Connection reset by peer (error 104)
2026-08-12T02:35:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:00 dist-test-slave-vq2q krb5kdc[4966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:00 dist-test-slave-vq2q krb5kdc[4966](info): set up 2 sockets
Aug 12 02:35:00 dist-test-slave-vq2q krb5kdc[4966](info): commencing operation
krb5kdc: starting...
W20260812 02:35:01.957736 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.007s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:02 dist-test-slave-vq2q krb5kdc[4966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:02.298791 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:02.670044 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:02.670377 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:02.670527 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:02.677744 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:02.677881 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:02.677965 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:02.678045 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:02.678105 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:02.688431 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:02.689888 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:02.691248 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:02.703373 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:02.703285 4992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:02.703338 4994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:02.704593 4985 server_base.cc:1034] running on GCE node
I20260812 02:35:02.705456 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:02.706931 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:02.708137 4985 hybrid_clock.cc:648] HybridClock initialized: now 1786502102708094 us; error 39 us; skew 500 ppm
I20260812 02:35:02.708492 4985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:02 dist-test-slave-vq2q krb5kdc[4966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:02.711634 4985 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:02.713253 4985 webserver.cc:460] Webserver started at http://127.0.0.1:33065/ using document root <none> and password file <none>
I20260812 02:35:02.713914 4985 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:02.714015 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:02.714264 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:02.717360 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "ac5651cc772f450fab4edcad7ceb035f"
format_stamp: "Formatted at 2026-08-12 02:35:02 on dist-test-slave-vq2q"
server_key: "91b303339d22f617c0b1ad9b1a441b29"
server_key_iv: "1c2d60740af231ef81ed5fe1dd0de310"
server_key_version: "encryptionkey@0"
I20260812 02:35:02.718107 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "ac5651cc772f450fab4edcad7ceb035f"
format_stamp: "Formatted at 2026-08-12 02:35:02 on dist-test-slave-vq2q"
server_key: "91b303339d22f617c0b1ad9b1a441b29"
server_key_iv: "1c2d60740af231ef81ed5fe1dd0de310"
server_key_version: "encryptionkey@0"
I20260812 02:35:02.723302 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 02:35:02.726845 5001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:02.728185 4985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:35:02.728332 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "ac5651cc772f450fab4edcad7ceb035f"
format_stamp: "Formatted at 2026-08-12 02:35:02 on dist-test-slave-vq2q"
server_key: "91b303339d22f617c0b1ad9b1a441b29"
server_key_iv: "1c2d60740af231ef81ed5fe1dd0de310"
server_key_version: "encryptionkey@0"
I20260812 02:35:02.728492 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:02.747956 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:02.752043 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:02.752285 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:02.770447 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43697
I20260812 02:35:02.770488 5061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43697 every 8 connection(s)
I20260812 02:35:02.771806 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:02.772552 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 4985
I20260812 02:35:02.772744 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:02.773335 12323 external_mini_cluster.cc:1428] Setting key bb992919b708dc3dea9b87b1306e3103
I20260812 02:35:02.776153 5062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:02.788380 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:02.775210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48681 (local address 127.0.0.1:43697)
0812 02:35:02.775994 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:02.776009 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:02.776059 (+ 50us) server_negotiation.cc:407] Connection header received
0812 02:35:02.778331 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:02.778356 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:02.778818 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:02.779253 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:02.779741 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:02.781751 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:02.783098 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:02.784106 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:02.784289 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:02.785546 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:02.785557 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:02.785580 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:02.786348 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:02.786450 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:02.786481 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:02.786650 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:02.786915 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 02:35:02.787093 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":329,"threads_started":1}
I20260812 02:35:02.789958 5062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f: Bootstrap starting.
I20260812 02:35:02.792516 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:02.793283 5062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:02.794255 5062 log.cc:824] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:02.797468 5062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f: No bootstrap required, opened a new log
I20260812 02:35:02.803807 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:02.797044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43235 (local address 127.0.0.1:43697)
0812 02:35:02.797223 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:02.797229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:02.797780 (+ 551us) server_negotiation.cc:407] Connection header received
0812 02:35:02.797946 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:02.797952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:02.798029 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:02.798135 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:02.798621 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:02.799763 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:02.801083 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:02.801769 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:02.801892 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:02.802945 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:02.802951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:02.802957 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:02.802980 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:02.803047 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:02.803071 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:02.803189 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:02.803417 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 02:35:02.803475 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 02:35:02.806403 5062 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5651cc772f450fab4edcad7ceb035f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } }
I20260812 02:35:02.806744 5062 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:02.806826 5062 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5651cc772f450fab4edcad7ceb035f, State: Initialized, Role: FOLLOWER
I20260812 02:35:02.807314 5062 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5651cc772f450fab4edcad7ceb035f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } }
I20260812 02:35:02.807519 5062 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:02.807615 5062 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:02.807749 5062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:02.809958 5062 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5651cc772f450fab4edcad7ceb035f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } }
I20260812 02:35:02.810429 5062 leader_election.cc:302] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac5651cc772f450fab4edcad7ceb035f; no voters:
I20260812 02:35:02.811066 5062 leader_election.cc:288] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:02.811173 5072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:02.812269 5072 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [term 1 LEADER]: Becoming Leader. State: Replica: ac5651cc772f450fab4edcad7ceb035f, State: Running, Role: LEADER
I20260812 02:35:02.812855 5072 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5651cc772f450fab4edcad7ceb035f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } }
I20260812 02:35:02.813211 5062 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:02.815472 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac5651cc772f450fab4edcad7ceb035f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5651cc772f450fab4edcad7ceb035f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } } }
I20260812 02:35:02.815863 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:02.817313 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac5651cc772f450fab4edcad7ceb035f. Latest consensus state: current_term: 1 leader_uuid: "ac5651cc772f450fab4edcad7ceb035f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5651cc772f450fab4edcad7ceb035f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } } }
I20260812 02:35:02.817700 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:02.818801 5079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:02.824324 5079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:02.833232 5079 catalog_manager.cc:1347] Generated new cluster ID: fcb31422664b47108c6d9dede161f132
I20260812 02:35:02.833359 5079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:02.859593 5079 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:02.861003 5079 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:02.867561 5079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac5651cc772f450fab4edcad7ceb035f: Generated new TSK 0
I20260812 02:35:02.868356 5079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:02.875586 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 4985
W20260812 02:35:02.876708 5057 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48681
W20260812 02:35:02.877593 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 4985: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:02.877787 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 4985
W20260812 02:35:02.899204 4964 connection.cc:535] client connection to 127.0.0.1:43697 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43697): Connection reset by peer (error 104)
2026-08-12T02:35:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2071 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:02 dist-test-slave-vq2q krb5kdc[5096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:02 dist-test-slave-vq2q krb5kdc[5096](info): set up 2 sockets
Aug 12 02:35:02 dist-test-slave-vq2q krb5kdc[5096](info): commencing operation
krb5kdc: starting...
W20260812 02:35:04.033170 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:04 dist-test-slave-vq2q krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:04.381512 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:04.741293 5112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:04.741639 5112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:04.741794 5112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:04.749342 5112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:04.749473 5112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:04.749547 5112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:04.749619 5112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:04.749656 5112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:04.760073 5112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:04.761493 5112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:04.762807 5112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:04.774444 5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:04.774497 5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:04.774657 5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:04.775364 5112 server_base.cc:1034] running on GCE node
I20260812 02:35:04.776108 5112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:04.777500 5112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:04.778699 5112 hybrid_clock.cc:648] HybridClock initialized: now 1786502104778662 us; error 26 us; skew 500 ppm
I20260812 02:35:04.779011 5112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:04 dist-test-slave-vq2q krb5kdc[5096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:04.782042 5112 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:04.783445 5112 webserver.cc:460] Webserver started at http://127.0.0.1:36137/ using document root <none> and password file <none>
I20260812 02:35:04.784073 5112 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:04.784178 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:04.784443 5112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:04.787549 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "f123c3debc59443590eb856d8c3e505e"
format_stamp: "Formatted at 2026-08-12 02:35:04 on dist-test-slave-vq2q"
server_key: "0ad9569b817ceaaac8645131905a402e"
server_key_iv: "345d53f172ae42a7827f09b5ad6fecb9"
server_key_version: "encryptionkey@0"
I20260812 02:35:04.788301 5112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "f123c3debc59443590eb856d8c3e505e"
format_stamp: "Formatted at 2026-08-12 02:35:04 on dist-test-slave-vq2q"
server_key: "0ad9569b817ceaaac8645131905a402e"
server_key_iv: "345d53f172ae42a7827f09b5ad6fecb9"
server_key_version: "encryptionkey@0"
I20260812 02:35:04.793569 5112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 02:35:04.797027 5129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:04.798324 5112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:35:04.798483 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "f123c3debc59443590eb856d8c3e505e"
format_stamp: "Formatted at 2026-08-12 02:35:04 on dist-test-slave-vq2q"
server_key: "0ad9569b817ceaaac8645131905a402e"
server_key_iv: "345d53f172ae42a7827f09b5ad6fecb9"
server_key_version: "encryptionkey@0"
I20260812 02:35:04.798671 5112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:04.804418 5112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:04.808670 5112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:04.808950 5112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:04.827891 5112 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40525
I20260812 02:35:04.827947 5192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40525 every 8 connection(s)
I20260812 02:35:04.829380 5112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:04.833238 5193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:04.837899 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 5112
I20260812 02:35:04.838099 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:04.838693 12323 external_mini_cluster.cc:1428] Setting key 20f37cb1ab56c080e24e7b1bba706a04
I20260812 02:35:04.847441 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:04.840651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38929 (local address 127.0.0.1:40525)
0812 02:35:04.841522 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:04.841537 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:04.841594 (+ 57us) server_negotiation.cc:407] Connection header received
0812 02:35:04.843376 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:04.843409 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:04.843917 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:04.844387 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:04.845441 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:04.845450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:04.845466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:04.845485 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:04.845561 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:04.845588 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:04.845837 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:04.845977 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 02:35:04.846166 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":335,"threads_started":1}
I20260812 02:35:04.848325 5193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e: Bootstrap starting.
I20260812 02:35:04.851501 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:04.851787 5193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:04.852860 5193 log.cc:824] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:04.856313 5193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e: No bootstrap required, opened a new log
I20260812 02:35:04.858634 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:04.856052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34923 (local address 127.0.0.1:40525)
0812 02:35:04.856205 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:04.856209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:04.856609 (+ 400us) server_negotiation.cc:407] Connection header received
0812 02:35:04.856766 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:04.856770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:04.856863 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:04.856955 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:04.857954 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:04.857959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:04.857962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:04.857979 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:04.858024 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:04.858040 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:04.858137 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:04.858266 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 02:35:04.858320 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 02:35:04.864666 5193 raft_consensus.cc:348] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123c3debc59443590eb856d8c3e505e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40525 } }
I20260812 02:35:04.865106 5193 raft_consensus.cc:374] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:04.865238 5193 raft_consensus.cc:729] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f123c3debc59443590eb856d8c3e505e, State: Initialized, Role: FOLLOWER
I20260812 02:35:04.865866 5193 consensus_queue.cc:260] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123c3debc59443590eb856d8c3e505e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40525 } }
I20260812 02:35:04.866107 5193 raft_consensus.cc:388] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:04.866242 5193 raft_consensus.cc:482] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:04.866392 5193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:04.868716 5193 raft_consensus.cc:504] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123c3debc59443590eb856d8c3e505e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40525 } }
I20260812 02:35:04.869328 5193 leader_election.cc:302] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f123c3debc59443590eb856d8c3e505e; no voters:
I20260812 02:35:04.869961 5193 leader_election.cc:288] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:04.870114 5205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:04.871318 5205 raft_consensus.cc:686] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [term 1 LEADER]: Becoming Leader. State: Replica: f123c3debc59443590eb856d8c3e505e, State: Running, Role: LEADER
I20260812 02:35:04.871934 5205 consensus_queue.cc:237] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123c3debc59443590eb856d8c3e505e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40525 } }
I20260812 02:35:04.872236 5193 sys_catalog.cc:564] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:04.875452 5207 sys_catalog.cc:455] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f123c3debc59443590eb856d8c3e505e. Latest consensus state: current_term: 1 leader_uuid: "f123c3debc59443590eb856d8c3e505e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123c3debc59443590eb856d8c3e505e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40525 } } }
I20260812 02:35:04.875445 5206 sys_catalog.cc:455] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f123c3debc59443590eb856d8c3e505e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123c3debc59443590eb856d8c3e505e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40525 } } }
I20260812 02:35:04.875947 5206 sys_catalog.cc:458] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:04.875948 5207 sys_catalog.cc:458] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:04.877456 5213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:04.883074 5213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:04.891963 5213 catalog_manager.cc:1347] Generated new cluster ID: e89e5e888e6448b8bef9fa0e6ae09315
I20260812 02:35:04.892092 5213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:04.917083 5213 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:04.917639 5213 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:04.928469 5213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f123c3debc59443590eb856d8c3e505e: Generated new TSK 0
I20260812 02:35:04.929555 5213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:04.943701 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5112
W20260812 02:35:04.944805 5186 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38929
W20260812 02:35:04.945567 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5112: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:04.945732 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 5112
2026-08-12T02:35:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:05 dist-test-slave-vq2q krb5kdc[5229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:05 dist-test-slave-vq2q krb5kdc[5229](info): set up 2 sockets
Aug 12 02:35:05 dist-test-slave-vq2q krb5kdc[5229](info): commencing operation
krb5kdc: starting...
W20260812 02:35:06.131255 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:06 dist-test-slave-vq2q krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:06.589687 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:07.025430 5242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:07.025768 5242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:07.025870 5242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:07.033208 5242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:07.033353 5242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:07.033422 5242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:07.033490 5242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:07.033540 5242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:07.043635 5242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:07.044947 5242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:07.046451 5242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:07.060513 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:07.060513 5251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:07.060567 5254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:07.062640 5242 server_base.cc:1034] running on GCE node
I20260812 02:35:07.063589 5242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:07.065302 5242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:07.066583 5242 hybrid_clock.cc:648] HybridClock initialized: now 1786502107066519 us; error 58 us; skew 500 ppm
I20260812 02:35:07.067085 5242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:07 dist-test-slave-vq2q krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:07.075683 5242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:07.077718 5242 webserver.cc:460] Webserver started at http://127.0.0.1:40367/ using document root <none> and password file <none>
I20260812 02:35:07.078497 5242 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:07.078632 5242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:07.078955 5242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:07.082245 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "e6a5e09362fd4edda4650231c0d1c732"
format_stamp: "Formatted at 2026-08-12 02:35:07 on dist-test-slave-vq2q"
server_key: "0a21a8807c609d6a1aaaf88ab5f28140"
server_key_iv: "457a49c5179f60e8c40006e90621f6f1"
server_key_version: "encryptionkey@0"
I20260812 02:35:07.083040 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "e6a5e09362fd4edda4650231c0d1c732"
format_stamp: "Formatted at 2026-08-12 02:35:07 on dist-test-slave-vq2q"
server_key: "0a21a8807c609d6a1aaaf88ab5f28140"
server_key_iv: "457a49c5179f60e8c40006e90621f6f1"
server_key_version: "encryptionkey@0"
I20260812 02:35:07.089435 5242 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 02:35:07.093667 5263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:07.095328 5242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:35:07.095536 5242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "e6a5e09362fd4edda4650231c0d1c732"
format_stamp: "Formatted at 2026-08-12 02:35:07 on dist-test-slave-vq2q"
server_key: "0a21a8807c609d6a1aaaf88ab5f28140"
server_key_iv: "457a49c5179f60e8c40006e90621f6f1"
server_key_version: "encryptionkey@0"
I20260812 02:35:07.095743 5242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:07.109692 5242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:07.114218 5242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:07.114501 5242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:07.135596 5242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39039
I20260812 02:35:07.135718 5324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39039 every 8 connection(s)
I20260812 02:35:07.137198 5242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:07.140272 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 5242
I20260812 02:35:07.140503 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:07.141151 12323 external_mini_cluster.cc:1428] Setting key 200b82aa564ab7403080d2a09fd8ab6a
I20260812 02:35:07.142660 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 02:35:07 dist-test-slave-vq2q krb5kdc[5229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:35:07.160063 5325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732: Bootstrap starting.
I20260812 02:35:07.164757 5325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:07.166422 5325 log.cc:824] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:07.168990 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:07.143691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60733 (local address 127.0.0.1:39039)
0812 02:35:07.144652 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:07.144667 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:07.144721 (+ 54us) server_negotiation.cc:407] Connection header received
0812 02:35:07.146592 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:07.146621 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:07.147109 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:07.147544 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:07.149916 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:07.151441 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:07.153476 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:07.154252 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:07.154413 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:07.160245 (+ 5832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:07.160257 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:07.160281 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:07.160340 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:07.164034 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:07.164864 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:07.164871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:07.164878 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:07.165104 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:07.165548 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:07.165554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:07.165558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:07.166385 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:07.166704 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:07.167354 (+ 650us) server_negotiation.cc:299] Negotiation successful
0812 02:35:07.167835 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":438,"threads_started":1}
I20260812 02:35:07.170466 5325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732: No bootstrap required, opened a new log
I20260812 02:35:07.173331 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:07.181979 5325 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a5e09362fd4edda4650231c0d1c732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } }
I20260812 02:35:07.182662 5325 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:07.182791 5325 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a5e09362fd4edda4650231c0d1c732, State: Initialized, Role: FOLLOWER
I20260812 02:35:07.183653 5325 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a5e09362fd4edda4650231c0d1c732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } }
I20260812 02:35:07.183931 5325 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:07.184113 5325 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:07.184329 5325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:07.187984 5325 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a5e09362fd4edda4650231c0d1c732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } }
I20260812 02:35:07.188889 5325 leader_election.cc:302] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6a5e09362fd4edda4650231c0d1c732; no voters:
I20260812 02:35:07.189839 5325 leader_election.cc:288] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:07.190021 5340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:07.191596 5340 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [term 1 LEADER]: Becoming Leader. State: Replica: e6a5e09362fd4edda4650231c0d1c732, State: Running, Role: LEADER
I20260812 02:35:07.192425 5340 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a5e09362fd4edda4650231c0d1c732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } }
I20260812 02:35:07.193301 5325 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:07.196122 5341 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6a5e09362fd4edda4650231c0d1c732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a5e09362fd4edda4650231c0d1c732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } } }
I20260812 02:35:07.196607 5341 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:07.199632 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:07.179947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33337 (local address 127.0.0.1:39039)
0812 02:35:07.180183 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:07.180189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:07.180617 (+ 428us) server_negotiation.cc:407] Connection header received
0812 02:35:07.180817 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:07.180821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:07.180895 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:07.181001 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:07.183264 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:07.184093 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:07.186076 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:07.186631 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:07.186751 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:07.190261 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:07.190268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:07.190272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:07.190315 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:07.194857 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:07.195929 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:07.195935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:07.195938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:07.196131 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:07.196634 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:07.196640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:07.196643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:07.197092 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:07.197432 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:07.198956 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0812 02:35:07.199220 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 02:35:07.202106 5342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6a5e09362fd4edda4650231c0d1c732. Latest consensus state: current_term: 1 leader_uuid: "e6a5e09362fd4edda4650231c0d1c732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6a5e09362fd4edda4650231c0d1c732" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } } }
I20260812 02:35:07.202401 5342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:07.206738 5345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:07.214506 5345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:07.226148 5345 catalog_manager.cc:1347] Generated new cluster ID: f4dfaffb334a42a39d85131fec281b74
I20260812 02:35:07.226337 5345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:07.235893 5345 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:07.237594 5345 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:07.248190 5345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6a5e09362fd4edda4650231c0d1c732: Generated new TSK 0
I20260812 02:35:07.249445 5345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:07.264748 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5242
I20260812 02:35:07.266541 5320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:07.374271 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 5242
2026-08-12T02:35:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:07 dist-test-slave-vq2q krb5kdc[5363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:07 dist-test-slave-vq2q krb5kdc[5363](info): set up 2 sockets
Aug 12 02:35:07 dist-test-slave-vq2q krb5kdc[5363](info): commencing operation
krb5kdc: starting...
W20260812 02:35:08.590669 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.002s sys 0.076s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:08 dist-test-slave-vq2q krb5kdc[5363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:08.940783 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:09.366758 5379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:09.367167 5379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:09.367318 5379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:09.375819 5379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:09.376025 5379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:09.376117 5379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:09.376214 5379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:09.376276 5379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:09.388192 5379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:09.389983 5379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:09.391642 5379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:09.406050 5386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:09.406013 5387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:09.405993 5389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:09.408254 5379 server_base.cc:1034] running on GCE node
I20260812 02:35:09.409531 5379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:09.411557 5379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:09.412807 5379 hybrid_clock.cc:648] HybridClock initialized: now 1786502109412760 us; error 45 us; skew 500 ppm
I20260812 02:35:09.413280 5379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:09 dist-test-slave-vq2q krb5kdc[5363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:09.417549 5379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:09.419660 5379 webserver.cc:460] Webserver started at http://127.0.0.1:36085/ using document root <none> and password file <none>
I20260812 02:35:09.420569 5379 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:09.420706 5379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:09.421088 5379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:09.425151 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "40476d6105064f59a56e91d945c6bb6f"
format_stamp: "Formatted at 2026-08-12 02:35:09 on dist-test-slave-vq2q"
server_key: "0b3d66614f4d208dc49414f76089e1d6"
server_key_iv: "0439ac67b7f9d92ea8d77b9e056b6beb"
server_key_version: "encryptionkey@0"
I20260812 02:35:09.426277 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "40476d6105064f59a56e91d945c6bb6f"
format_stamp: "Formatted at 2026-08-12 02:35:09 on dist-test-slave-vq2q"
server_key: "0b3d66614f4d208dc49414f76089e1d6"
server_key_iv: "0439ac67b7f9d92ea8d77b9e056b6beb"
server_key_version: "encryptionkey@0"
I20260812 02:35:09.432816 5379 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 02:35:09.437050 5399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:09.438975 5379 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:35:09.439208 5379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "40476d6105064f59a56e91d945c6bb6f"
format_stamp: "Formatted at 2026-08-12 02:35:09 on dist-test-slave-vq2q"
server_key: "0b3d66614f4d208dc49414f76089e1d6"
server_key_iv: "0439ac67b7f9d92ea8d77b9e056b6beb"
server_key_version: "encryptionkey@0"
I20260812 02:35:09.439477 5379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:09.454874 5379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:09.460760 5379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:09.461128 5379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:09.482635 5379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37197
I20260812 02:35:09.482659 5462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37197 every 8 connection(s)
I20260812 02:35:09.484447 5379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:09.489652 5464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:09.489871 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 5379
I20260812 02:35:09.490085 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:09.490736 12323 external_mini_cluster.cc:1428] Setting key 21174c4b65670aa7eebe3edd4aa3cbfc
I20260812 02:35:09.506912 5464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f: Bootstrap starting.
I20260812 02:35:09.506953 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:09.493041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57385 (local address 127.0.0.1:37197)
0812 02:35:09.494145 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:09.494161 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:09.494216 (+ 55us) server_negotiation.cc:407] Connection header received
0812 02:35:09.495982 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:09.496024 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:09.496607 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:09.497060 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:09.497792 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:09.499302 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:09.501285 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:09.502142 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:09.502329 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:09.503784 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:09.503796 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:09.503817 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:09.504583 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:09.504688 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:09.504719 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:09.504826 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:09.505071 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 02:35:09.505505 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":459,"threads_started":1}
I20260812 02:35:09.511618 5464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:09.512292 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:09.513248 5464 log.cc:824] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:09.517869 5464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f: No bootstrap required, opened a new log
I20260812 02:35:09.525977 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:09.518324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57009 (local address 127.0.0.1:37197)
0812 02:35:09.518602 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:09.518616 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:09.519085 (+ 469us) server_negotiation.cc:407] Connection header received
0812 02:35:09.519286 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:09.519291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:09.519381 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:09.519480 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:09.520205 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:09.521029 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:09.522993 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:09.523566 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:09.523690 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:09.525099 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:09.525104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:09.525108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:09.525132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:09.525234 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:09.525257 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:09.525344 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:09.525490 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 02:35:09.525595 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:35:09.529260 5464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40476d6105064f59a56e91d945c6bb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } }
I20260812 02:35:09.529811 5464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:09.529918 5464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40476d6105064f59a56e91d945c6bb6f, State: Initialized, Role: FOLLOWER
I20260812 02:35:09.530609 5464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40476d6105064f59a56e91d945c6bb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } }
I20260812 02:35:09.530872 5464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:09.531016 5464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:09.531198 5464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:09.534140 5464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40476d6105064f59a56e91d945c6bb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } }
I20260812 02:35:09.534829 5464 leader_election.cc:302] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40476d6105064f59a56e91d945c6bb6f; no voters:
I20260812 02:35:09.535564 5464 leader_election.cc:288] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:09.535781 5476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:09.537086 5476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [term 1 LEADER]: Becoming Leader. State: Replica: 40476d6105064f59a56e91d945c6bb6f, State: Running, Role: LEADER
I20260812 02:35:09.537794 5476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40476d6105064f59a56e91d945c6bb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } }
I20260812 02:35:09.538326 5464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:09.540947 5478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40476d6105064f59a56e91d945c6bb6f. Latest consensus state: current_term: 1 leader_uuid: "40476d6105064f59a56e91d945c6bb6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40476d6105064f59a56e91d945c6bb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } } }
I20260812 02:35:09.541445 5478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:09.542055 5477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40476d6105064f59a56e91d945c6bb6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40476d6105064f59a56e91d945c6bb6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } } }
I20260812 02:35:09.542418 5477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:09.545413 5488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:09.550637 5488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:09.561455 5488 catalog_manager.cc:1347] Generated new cluster ID: b53b0f75bbe3488094fdb0216238b7df
I20260812 02:35:09.561630 5488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:09.592384 5488 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:09.594216 5488 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:09.603294 5488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40476d6105064f59a56e91d945c6bb6f: Generated new TSK 0
I20260812 02:35:09.604488 5488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:09.619858 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5379
W20260812 02:35:09.621348 5456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57385
W20260812 02:35:09.622165 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5379: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:09.622360 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 5379
2026-08-12T02:35:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:09 dist-test-slave-vq2q krb5kdc[5501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:09 dist-test-slave-vq2q krb5kdc[5501](info): set up 2 sockets
Aug 12 02:35:09 dist-test-slave-vq2q krb5kdc[5501](info): commencing operation
krb5kdc: starting...
W20260812 02:35:10.848353 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.001s sys 0.080s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:11 dist-test-slave-vq2q krb5kdc[5501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:11.297746 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:11.754534 5519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:11.754971 5519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:11.755082 5519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:11.763427 5519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:11.763607 5519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:11.763659 5519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:11.763727 5519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:11.763774 5519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:11.775825 5519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:11.778126 5519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:11.780614 5519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:11.798177 5524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:11.798233 5525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:11.798197 5527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:11.798596 5519 server_base.cc:1034] running on GCE node
I20260812 02:35:11.800195 5519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:11.802397 5519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:11.803691 5519 hybrid_clock.cc:648] HybridClock initialized: now 1786502111803629 us; error 61 us; skew 500 ppm
I20260812 02:35:11.804180 5519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:11 dist-test-slave-vq2q krb5kdc[5501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:11.808452 5519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:11.810556 5519 webserver.cc:460] Webserver started at http://127.0.0.1:45791/ using document root <none> and password file <none>
I20260812 02:35:11.811525 5519 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:11.811681 5519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:11.812085 5519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:11.816062 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "2343f8dadcff4c418e5b43631537b527"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-vq2q"
server_key: "02cc46ee94181480969413ac242b4252"
server_key_iv: "683896d7fbb8366b9190c8ef437c0f57"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.817101 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "2343f8dadcff4c418e5b43631537b527"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-vq2q"
server_key: "02cc46ee94181480969413ac242b4252"
server_key_iv: "683896d7fbb8366b9190c8ef437c0f57"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.824019 5519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:35:11.828989 5537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.830832 5519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:35:11.831066 5519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "2343f8dadcff4c418e5b43631537b527"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-vq2q"
server_key: "02cc46ee94181480969413ac242b4252"
server_key_iv: "683896d7fbb8366b9190c8ef437c0f57"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.831359 5519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:11.838945 5519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:11.844480 5519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:11.844861 5519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:11.867269 5519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41563
I20260812 02:35:11.867285 5593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41563 every 8 connection(s)
I20260812 02:35:11.869153 5519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:11.874444 5596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:11.877820 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 5519
I20260812 02:35:11.878079 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:11.878706 12323 external_mini_cluster.cc:1428] Setting key 28e66cc4be323eaabcbe39860e016878
I20260812 02:35:11.889595 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:11.881400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52555 (local address 127.0.0.1:41563)
0812 02:35:11.882472 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:11.882488 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:11.882547 (+ 59us) server_negotiation.cc:407] Connection header received
0812 02:35:11.884669 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:11.884702 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:11.885255 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:11.885821 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:11.887246 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:11.887254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:11.887277 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:11.887296 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:11.887386 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:11.887416 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:11.887725 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:11.887914 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 02:35:11.888183 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":478,"threads_started":1}
I20260812 02:35:11.890630 5596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527: Bootstrap starting.
I20260812 02:35:11.894452 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:11.894907 5596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:11.896272 5596 log.cc:824] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:11.900893 5596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527: No bootstrap required, opened a new log
I20260812 02:35:11.902832 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:11.899684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49687 (local address 127.0.0.1:41563)
0812 02:35:11.899914 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:11.899923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:11.900408 (+ 485us) server_negotiation.cc:407] Connection header received
0812 02:35:11.900585 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:11.900591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:11.900672 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:11.900819 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:11.901968 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:11.901976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:11.901982 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:11.902011 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:11.902088 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:11.902119 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:11.902289 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:11.902353 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 02:35:11.902437 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:35:11.911566 5596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2343f8dadcff4c418e5b43631537b527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } }
I20260812 02:35:11.912111 5596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:11.912196 5596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2343f8dadcff4c418e5b43631537b527, State: Initialized, Role: FOLLOWER
I20260812 02:35:11.912883 5596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2343f8dadcff4c418e5b43631537b527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } }
I20260812 02:35:11.913132 5596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:11.913302 5596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:11.913507 5596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:11.916577 5596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2343f8dadcff4c418e5b43631537b527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } }
I20260812 02:35:11.917392 5596 leader_election.cc:302] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2343f8dadcff4c418e5b43631537b527; no voters:
I20260812 02:35:11.918182 5596 leader_election.cc:288] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:11.918406 5606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:11.919140 5606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [term 1 LEADER]: Becoming Leader. State: Replica: 2343f8dadcff4c418e5b43631537b527, State: Running, Role: LEADER
I20260812 02:35:11.919791 5606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2343f8dadcff4c418e5b43631537b527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } }
I20260812 02:35:11.921090 5596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:11.923635 5608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2343f8dadcff4c418e5b43631537b527. Latest consensus state: current_term: 1 leader_uuid: "2343f8dadcff4c418e5b43631537b527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2343f8dadcff4c418e5b43631537b527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } } }
I20260812 02:35:11.924399 5608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:11.929508 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2343f8dadcff4c418e5b43631537b527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2343f8dadcff4c418e5b43631537b527" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41563 } } }
I20260812 02:35:11.930243 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:11.932260 5613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:11.939437 5613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:11.951509 5613 catalog_manager.cc:1347] Generated new cluster ID: 671aa56659824ca4a0fe22ae96dd2af0
I20260812 02:35:11.951692 5613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:11.998678 5613 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:11.999341 5613 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:12.011322 5613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2343f8dadcff4c418e5b43631537b527: Generated new TSK 0
I20260812 02:35:12.012493 5613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:12.023221 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5519
W20260812 02:35:12.025247 5589 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52555
W20260812 02:35:12.026315 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5519: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:12.026513 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 5519
2026-08-12T02:35:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:12 dist-test-slave-vq2q krb5kdc[5632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:12 dist-test-slave-vq2q krb5kdc[5632](info): set up 2 sockets
Aug 12 02:35:12 dist-test-slave-vq2q krb5kdc[5632](info): commencing operation
krb5kdc: starting...
W20260812 02:35:13.276392 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.079s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:13 dist-test-slave-vq2q krb5kdc[5632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:13.706988 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:14.150668 5648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:14.151031 5648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:14.151140 5648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:14.158663 5648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:14.158812 5648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:14.158861 5648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:14.158937 5648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:14.158986 5648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:14.169329 5648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:14.170769 5648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:14.172302 5648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:14.185817 5658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:14.185840 5656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:14.185933 5655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:14.187505 5648 server_base.cc:1034] running on GCE node
I20260812 02:35:14.188560 5648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:14.190536 5648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:14.191766 5648 hybrid_clock.cc:648] HybridClock initialized: now 1786502114191696 us; error 67 us; skew 500 ppm
I20260812 02:35:14.192193 5648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:14 dist-test-slave-vq2q krb5kdc[5632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:14.195976 5648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:14.197908 5648 webserver.cc:460] Webserver started at http://127.0.0.1:45461/ using document root <none> and password file <none>
I20260812 02:35:14.198763 5648 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:14.198889 5648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:14.199229 5648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:14.203346 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "b9ce20bd55b44ca7900aa09c1d4548e2"
format_stamp: "Formatted at 2026-08-12 02:35:14 on dist-test-slave-vq2q"
server_key: "c37383a2414993abdc0bf75f63d37687"
server_key_iv: "98536f0a9577ea23b63fab991b94d65f"
server_key_version: "encryptionkey@0"
I20260812 02:35:14.204356 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "b9ce20bd55b44ca7900aa09c1d4548e2"
format_stamp: "Formatted at 2026-08-12 02:35:14 on dist-test-slave-vq2q"
server_key: "c37383a2414993abdc0bf75f63d37687"
server_key_iv: "98536f0a9577ea23b63fab991b94d65f"
server_key_version: "encryptionkey@0"
I20260812 02:35:14.211236 5648 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260812 02:35:14.216003 5667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:14.218034 5648 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:35:14.218276 5648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "b9ce20bd55b44ca7900aa09c1d4548e2"
format_stamp: "Formatted at 2026-08-12 02:35:14 on dist-test-slave-vq2q"
server_key: "c37383a2414993abdc0bf75f63d37687"
server_key_iv: "98536f0a9577ea23b63fab991b94d65f"
server_key_version: "encryptionkey@0"
I20260812 02:35:14.218544 5648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:14.235090 5648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:14.240824 5648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:14.241226 5648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:14.266187 5648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38549
I20260812 02:35:14.266251 5726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38549 every 8 connection(s)
I20260812 02:35:14.268101 5648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:14.272476 5727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:14.278218 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 5648
I20260812 02:35:14.278420 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:14.278936 12323 external_mini_cluster.cc:1428] Setting key e959a9886b63b981f621dd7549f95cad
I20260812 02:35:14.285624 5727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2: Bootstrap starting.
I20260812 02:35:14.289570 5727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:14.290782 5727 log.cc:824] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:14.294576 5727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2: No bootstrap required, opened a new log
Aug 12 02:35:14 dist-test-slave-vq2q krb5kdc[5632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:35:14.303996 5727 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ce20bd55b44ca7900aa09c1d4548e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } }
I20260812 02:35:14.304549 5727 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:14.304716 5727 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ce20bd55b44ca7900aa09c1d4548e2, State: Initialized, Role: FOLLOWER
I20260812 02:35:14.305433 5727 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ce20bd55b44ca7900aa09c1d4548e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } }
I20260812 02:35:14.305702 5727 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:14.305889 5727 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:14.306068 5727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:14.308739 5727 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ce20bd55b44ca7900aa09c1d4548e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } }
I20260812 02:35:14.309044 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:14.281362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:51337 (local address 127.0.0.1:38549)
0812 02:35:14.282650 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:14.282668 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:14.282731 (+ 63us) server_negotiation.cc:407] Connection header received
0812 02:35:14.285167 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:14.285237 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:14.285822 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:14.286316 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:14.288025 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:14.290065 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:14.291927 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:14.292770 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:14.292973 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:14.298394 (+ 5421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:14.298409 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:14.298429 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:14.298483 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:14.302985 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:14.304227 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:14.304236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:14.304245 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:14.304493 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:14.305220 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:14.305228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:14.305232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:14.306197 (+ 965us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:14.306611 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:14.307233 (+ 622us) server_negotiation.cc:299] Negotiation successful
0812 02:35:14.307681 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":535,"threads_started":1}
I20260812 02:35:14.309419 5727 leader_election.cc:302] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ce20bd55b44ca7900aa09c1d4548e2; no voters:
I20260812 02:35:14.310279 5727 leader_election.cc:288] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:14.310461 5734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:14.311764 5734 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [term 1 LEADER]: Becoming Leader. State: Replica: b9ce20bd55b44ca7900aa09c1d4548e2, State: Running, Role: LEADER
I20260812 02:35:14.312534 5734 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ce20bd55b44ca7900aa09c1d4548e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } }
I20260812 02:35:14.312744 5727 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:14.315142 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:14.322768 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ce20bd55b44ca7900aa09c1d4548e2. Latest consensus state: current_term: 1 leader_uuid: "b9ce20bd55b44ca7900aa09c1d4548e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ce20bd55b44ca7900aa09c1d4548e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } } }
I20260812 02:35:14.323385 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:14.324018 5735 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ce20bd55b44ca7900aa09c1d4548e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ce20bd55b44ca7900aa09c1d4548e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } } }
I20260812 02:35:14.324337 5735 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:14.347522 5746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:14.355382 5746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:14.363965 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:14.339878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:55975 (local address 127.0.0.1:38549)
0812 02:35:14.340106 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:14.340113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:14.344218 (+ 4105us) server_negotiation.cc:407] Connection header received
0812 02:35:14.345322 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:14.345328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:14.345440 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:14.345524 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:14.347930 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:14.349215 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:14.351070 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:14.351769 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:14.351881 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:14.355555 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:14.355564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:14.355569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:14.355613 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:14.359791 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:14.360912 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:14.360918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:14.360922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:14.361107 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:14.361748 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:14.361754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:14.361758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:14.362204 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:14.362504 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:14.363319 (+ 815us) server_negotiation.cc:299] Negotiation successful
0812 02:35:14.363533 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 02:35:14.380726 5746 catalog_manager.cc:1347] Generated new cluster ID: 62db31c5c39940bda616e1657c92ae0e
I20260812 02:35:14.380880 5746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:14.405572 5746 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:14.407321 5746 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:14.424201 5746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ce20bd55b44ca7900aa09c1d4548e2: Generated new TSK 0
I20260812 02:35:14.425544 5746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:14.439543 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5648
I20260812 02:35:14.441457 5721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:14.551791 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 5648
W20260812 02:35:14.576792 5630 connection.cc:535] client connection to 127.0.0.1:38549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38549): Connection reset by peer (error 104)
2026-08-12T02:35:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:14 dist-test-slave-vq2q krb5kdc[5764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:14 dist-test-slave-vq2q krb5kdc[5764](info): set up 2 sockets
Aug 12 02:35:14 dist-test-slave-vq2q krb5kdc[5764](info): commencing operation
krb5kdc: starting...
W20260812 02:35:15.757462 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.083s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:16 dist-test-slave-vq2q krb5kdc[5764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:16.186589 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:16.580715 5778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:16.581063 5778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:16.581238 5778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:16.588927 5778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:16.589082 5778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:16.589155 5778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:16.589284 5778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:16.589347 5778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:16.599977 5778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:16.601475 5778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:16.602844 5778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:16.615849 5783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:16.615962 5784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:16.615883 5787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:16.617460 5778 server_base.cc:1034] running on GCE node
I20260812 02:35:16.618407 5778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:16.620127 5778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:16.621377 5778 hybrid_clock.cc:648] HybridClock initialized: now 1786502116621314 us; error 61 us; skew 500 ppm
I20260812 02:35:16.621891 5778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:16 dist-test-slave-vq2q krb5kdc[5764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:16.625615 5778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:16.627631 5778 webserver.cc:460] Webserver started at http://127.0.0.1:38837/ using document root <none> and password file <none>
I20260812 02:35:16.628429 5778 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:16.628562 5778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:16.628906 5778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:16.632252 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "e51c40462e3f4dd6bf0e1869df99779d"
format_stamp: "Formatted at 2026-08-12 02:35:16 on dist-test-slave-vq2q"
server_key: "10e0efbfb474e5d57f84e59e72a44eff"
server_key_iv: "2ba106179a63c777ee315aec5ef8c4a0"
server_key_version: "encryptionkey@0"
I20260812 02:35:16.633071 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "e51c40462e3f4dd6bf0e1869df99779d"
format_stamp: "Formatted at 2026-08-12 02:35:16 on dist-test-slave-vq2q"
server_key: "10e0efbfb474e5d57f84e59e72a44eff"
server_key_iv: "2ba106179a63c777ee315aec5ef8c4a0"
server_key_version: "encryptionkey@0"
I20260812 02:35:16.639276 5778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260812 02:35:16.643795 5795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:16.645655 5778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 02:35:16.645929 5778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "e51c40462e3f4dd6bf0e1869df99779d"
format_stamp: "Formatted at 2026-08-12 02:35:16 on dist-test-slave-vq2q"
server_key: "10e0efbfb474e5d57f84e59e72a44eff"
server_key_iv: "2ba106179a63c777ee315aec5ef8c4a0"
server_key_version: "encryptionkey@0"
I20260812 02:35:16.646142 5778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:16.673974 5778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:16.678897 5778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:16.679188 5778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:16.700239 5778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43429
I20260812 02:35:16.700350 5856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43429 every 8 connection(s)
I20260812 02:35:16.701910 5778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:16.707454 5857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:16.712224 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 5778
I20260812 02:35:16.712471 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:16.713055 12323 external_mini_cluster.cc:1428] Setting key 3acac5959e5ecfff55aecfb4588e64d5
I20260812 02:35:16.723430 5857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d: Bootstrap starting.
I20260812 02:35:16.727861 5857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:16.728839 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:16.715553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:39809 (local address 127.0.0.1:43429)
0812 02:35:16.716590 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:16.716610 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:16.716665 (+ 55us) server_negotiation.cc:407] Connection header received
0812 02:35:16.718862 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:16.718897 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:16.719440 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:16.719878 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:16.720540 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:16.722042 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:16.723354 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:16.724113 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:16.724297 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:16.725784 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:16.725793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:16.725809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:16.726527 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:16.726765 (+ 238us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:16.726848 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:16.727039 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:16.727356 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 02:35:16.727550 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":423,"threads_started":1}
I20260812 02:35:16.729400 5857 log.cc:824] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:16.733465 5857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d: No bootstrap required, opened a new log
I20260812 02:35:16.733906 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:16.743299 5857 raft_consensus.cc:348] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51c40462e3f4dd6bf0e1869df99779d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } }
I20260812 02:35:16.743866 5857 raft_consensus.cc:374] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:16.743969 5857 raft_consensus.cc:729] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51c40462e3f4dd6bf0e1869df99779d, State: Initialized, Role: FOLLOWER
I20260812 02:35:16.744791 5857 consensus_queue.cc:260] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51c40462e3f4dd6bf0e1869df99779d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } }
I20260812 02:35:16.745111 5857 raft_consensus.cc:388] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:16.745332 5857 raft_consensus.cc:482] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:16.745570 5857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:16.746096 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:16.739219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:45253 (local address 127.0.0.1:43429)
0812 02:35:16.739433 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:16.739441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:16.739885 (+ 444us) server_negotiation.cc:407] Connection header received
0812 02:35:16.740080 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:16.740083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:16.740161 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:16.740250 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:16.740825 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:16.741710 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:16.743006 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:16.743466 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:16.743575 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:16.744951 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:16.744955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:16.744959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:16.744984 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:16.745050 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:16.745070 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:16.745255 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:16.745704 (+ 449us) server_negotiation.cc:299] Negotiation successful
0812 02:35:16.745783 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:35:16.748929 5857 raft_consensus.cc:504] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51c40462e3f4dd6bf0e1869df99779d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } }
I20260812 02:35:16.749711 5857 leader_election.cc:302] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e51c40462e3f4dd6bf0e1869df99779d; no voters:
I20260812 02:35:16.750944 5857 leader_election.cc:288] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:16.751291 5871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:16.752544 5871 raft_consensus.cc:686] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [term 1 LEADER]: Becoming Leader. State: Replica: e51c40462e3f4dd6bf0e1869df99779d, State: Running, Role: LEADER
I20260812 02:35:16.753304 5871 consensus_queue.cc:237] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51c40462e3f4dd6bf0e1869df99779d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } }
I20260812 02:35:16.755460 5857 sys_catalog.cc:564] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:16.757119 5872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e51c40462e3f4dd6bf0e1869df99779d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51c40462e3f4dd6bf0e1869df99779d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } } }
I20260812 02:35:16.757048 5873 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e51c40462e3f4dd6bf0e1869df99779d. Latest consensus state: current_term: 1 leader_uuid: "e51c40462e3f4dd6bf0e1869df99779d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51c40462e3f4dd6bf0e1869df99779d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } } }
I20260812 02:35:16.757704 5872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:16.757733 5873 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:16.761533 5876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:16.770036 5876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:16.782102 5876 catalog_manager.cc:1347] Generated new cluster ID: 67f7b5dee5eb4d87827ef7b19e6a277c
I20260812 02:35:16.782262 5876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:16.794602 5876 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:16.796342 5876 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:16.809720 5876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e51c40462e3f4dd6bf0e1869df99779d: Generated new TSK 0
I20260812 02:35:16.811467 5876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:16.821269 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5778
W20260812 02:35:16.822949 5851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.26:39809
W20260812 02:35:16.824246 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5778: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:16.824455 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 5778
W20260812 02:35:16.850746 5762 connection.cc:535] client connection to 127.0.0.1:43429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43429): Connection reset by peer (error 104)
2026-08-12T02:35:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:16 dist-test-slave-vq2q krb5kdc[5896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:16 dist-test-slave-vq2q krb5kdc[5896](info): set up 2 sockets
Aug 12 02:35:16 dist-test-slave-vq2q krb5kdc[5896](info): commencing operation
krb5kdc: starting...
W20260812 02:35:18.023656 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.004s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:18 dist-test-slave-vq2q krb5kdc[5896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:18.430382 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:18.795611 5910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:18.795930 5910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:18.796025 5910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:18.803427 5910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:18.803576 5910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:18.803627 5910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:18.803680 5910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:18.803730 5910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:18.814255 5910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:18.816111 5910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:18.817535 5910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:18.832080 5915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:18.832190 5916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:18.832230 5918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:18.833876 5910 server_base.cc:1034] running on GCE node
I20260812 02:35:18.835069 5910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:18.837083 5910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:18.838409 5910 hybrid_clock.cc:648] HybridClock initialized: now 1786502118838329 us; error 56 us; skew 500 ppm
I20260812 02:35:18.838896 5910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:18 dist-test-slave-vq2q krb5kdc[5896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:18.842947 5910 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:18.845083 5910 webserver.cc:460] Webserver started at http://127.0.0.1:33945/ using document root <none> and password file <none>
I20260812 02:35:18.846025 5910 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:18.846160 5910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:18.846493 5910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:18.850454 5910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "e367636992b846abbb5116942e374080"
format_stamp: "Formatted at 2026-08-12 02:35:18 on dist-test-slave-vq2q"
server_key: "391f0aef228b526268bf321d381182ed"
server_key_iv: "7f9c02d536a540de3e9e6965ea8e6f2e"
server_key_version: "encryptionkey@0"
I20260812 02:35:18.851423 5910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "e367636992b846abbb5116942e374080"
format_stamp: "Formatted at 2026-08-12 02:35:18 on dist-test-slave-vq2q"
server_key: "391f0aef228b526268bf321d381182ed"
server_key_iv: "7f9c02d536a540de3e9e6965ea8e6f2e"
server_key_version: "encryptionkey@0"
I20260812 02:35:18.858767 5910 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260812 02:35:18.863829 5927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:18.866029 5910 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:35:18.866302 5910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "e367636992b846abbb5116942e374080"
format_stamp: "Formatted at 2026-08-12 02:35:18 on dist-test-slave-vq2q"
server_key: "391f0aef228b526268bf321d381182ed"
server_key_iv: "7f9c02d536a540de3e9e6965ea8e6f2e"
server_key_version: "encryptionkey@0"
I20260812 02:35:18.866567 5910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:18.874310 5910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:18.879997 5910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:18.880412 5910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:18.903784 5910 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39937
I20260812 02:35:18.903825 5989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39937 every 8 connection(s)
I20260812 02:35:18.905879 5910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:18.910372 5990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:18.915383 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 5910
I20260812 02:35:18.915642 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:18.916177 12323 external_mini_cluster.cc:1428] Setting key 133520c508a1784842951837123ba8c7
I20260812 02:35:18.923311 5990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080: Bootstrap starting.
I20260812 02:35:18.927467 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:18.918875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:59725 (local address 127.0.0.1:39937)
0812 02:35:18.919975 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:18.919998 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:18.920062 (+ 64us) server_negotiation.cc:407] Connection header received
0812 02:35:18.922569 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:18.922604 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:18.923129 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:18.923656 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:18.925278 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:18.925288 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:18.925320 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:18.925345 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:18.925437 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:18.925472 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:18.925767 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:18.925922 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 02:35:18.926160 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":458,"threads_started":1}
I20260812 02:35:18.927706 5990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:18.929087 5990 log.cc:824] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:18.932895 5990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080: No bootstrap required, opened a new log
I20260812 02:35:18.933068 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:18.941811 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:18.938569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:45539 (local address 127.0.0.1:39937)
0812 02:35:18.938795 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:18.938802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:18.939236 (+ 434us) server_negotiation.cc:407] Connection header received
0812 02:35:18.939411 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:18.939427 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:18.939501 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:18.939662 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:18.940860 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:18.940865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:18.940870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:18.940891 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:18.940952 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:18.940972 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:18.941130 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:18.941395 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 02:35:18.941475 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 02:35:18.941533 5990 raft_consensus.cc:348] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e367636992b846abbb5116942e374080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } }
I20260812 02:35:18.942097 5990 raft_consensus.cc:374] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:18.942209 5990 raft_consensus.cc:729] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e367636992b846abbb5116942e374080, State: Initialized, Role: FOLLOWER
I20260812 02:35:18.942890 5990 consensus_queue.cc:260] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e367636992b846abbb5116942e374080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } }
I20260812 02:35:18.943171 5990 raft_consensus.cc:388] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:18.943322 5990 raft_consensus.cc:482] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:18.943513 5990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:18.945989 5990 raft_consensus.cc:504] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e367636992b846abbb5116942e374080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } }
I20260812 02:35:18.946527 5990 leader_election.cc:302] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e367636992b846abbb5116942e374080; no voters:
I20260812 02:35:18.947273 5990 leader_election.cc:288] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:18.947477 6000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:18.948459 6000 raft_consensus.cc:686] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [term 1 LEADER]: Becoming Leader. State: Replica: e367636992b846abbb5116942e374080, State: Running, Role: LEADER
I20260812 02:35:18.949232 6000 consensus_queue.cc:237] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e367636992b846abbb5116942e374080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } }
I20260812 02:35:18.949640 5990 sys_catalog.cc:564] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:18.952657 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e367636992b846abbb5116942e374080. Latest consensus state: current_term: 1 leader_uuid: "e367636992b846abbb5116942e374080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e367636992b846abbb5116942e374080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } } }
I20260812 02:35:18.953029 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e367636992b846abbb5116942e374080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e367636992b846abbb5116942e374080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39937 } } }
I20260812 02:35:18.953351 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:18.953509 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:18.955724 6008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:18.962343 6008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:18.974516 6008 catalog_manager.cc:1347] Generated new cluster ID: ecd9b5ffc093439596c697c0f526c63e
I20260812 02:35:18.974663 6008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:19.004024 6008 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:19.004527 6008 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:19.019675 6008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e367636992b846abbb5116942e374080: Generated new TSK 0
I20260812 02:35:19.020702 6008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:19.036338 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5910
W20260812 02:35:19.037945 5982 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.26:59725
W20260812 02:35:19.039031 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 5910: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:19.039215 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 5910
2026-08-12T02:35:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:19 dist-test-slave-vq2q krb5kdc[6023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:19 dist-test-slave-vq2q krb5kdc[6023](info): set up 2 sockets
Aug 12 02:35:19 dist-test-slave-vq2q krb5kdc[6023](info): commencing operation
krb5kdc: starting...
W20260812 02:35:20.234053 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.005s sys 0.071s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:20 dist-test-slave-vq2q krb5kdc[6023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:20.596866 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:20.955063 6039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:20.955380 6039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:20.955507 6039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:20.962957 6039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:20.963094 6039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:20.963176 6039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:20.963263 6039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:20.963325 6039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:20.973913 6039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:20.975363 6039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:20.976773 6039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:20.989374 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:20.989442 6045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:20.989411 6047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:20.990319 6039 server_base.cc:1034] running on GCE node
I20260812 02:35:20.991290 6039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:20.992882 6039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:20.994105 6039 hybrid_clock.cc:648] HybridClock initialized: now 1786502120994058 us; error 38 us; skew 500 ppm
I20260812 02:35:20.994479 6039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:20 dist-test-slave-vq2q krb5kdc[6023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:20.998014 6039 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:20.999730 6039 webserver.cc:460] Webserver started at http://127.0.0.1:44161/ using document root <none> and password file <none>
I20260812 02:35:21.000573 6039 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:21.000713 6039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:21.001044 6039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:21.004287 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "d68499eb3f8e4f419896900ead72d13d"
format_stamp: "Formatted at 2026-08-12 02:35:21 on dist-test-slave-vq2q"
server_key: "118bba861328caf003212065de580ee9"
server_key_iv: "2425ef76c9730ac206a82ce22f37f7e6"
server_key_version: "encryptionkey@0"
I20260812 02:35:21.005084 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "d68499eb3f8e4f419896900ead72d13d"
format_stamp: "Formatted at 2026-08-12 02:35:21 on dist-test-slave-vq2q"
server_key: "118bba861328caf003212065de580ee9"
server_key_iv: "2425ef76c9730ac206a82ce22f37f7e6"
server_key_version: "encryptionkey@0"
I20260812 02:35:21.010722 6039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 02:35:21.014567 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:21.015925 6039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:35:21.016105 6039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "d68499eb3f8e4f419896900ead72d13d"
format_stamp: "Formatted at 2026-08-12 02:35:21 on dist-test-slave-vq2q"
server_key: "118bba861328caf003212065de580ee9"
server_key_iv: "2425ef76c9730ac206a82ce22f37f7e6"
server_key_version: "encryptionkey@0"
I20260812 02:35:21.016304 6039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:21.051944 6039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:21.056392 6039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:21.056679 6039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:21.075173 6039 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37293
I20260812 02:35:21.075203 6115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37293 every 8 connection(s)
I20260812 02:35:21.076687 6039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:21.080556 6116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:21.083892 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 6039
I20260812 02:35:21.084076 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:21.084774 12323 external_mini_cluster.cc:1428] Setting key 3ba190ac3902e0da290b0a4ff47224c3
I20260812 02:35:21.093469 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d: Bootstrap starting.
I20260812 02:35:21.097432 6116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:21.098788 6116 log.cc:824] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:35:21 dist-test-slave-vq2q krb5kdc[6023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:35:21.102456 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d: No bootstrap required, opened a new log
I20260812 02:35:21.110809 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:21.086734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:59485 (local address 127.0.0.1:37293)
0812 02:35:21.087498 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:21.087510 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:21.087554 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:35:21.089319 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:21.089343 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:21.089766 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:21.090106 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:21.091954 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:21.093405 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:21.095191 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:21.095930 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:21.096070 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:21.101455 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:21.101469 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:21.101489 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:21.101537 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:21.105646 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:21.106603 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:21.106612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:21.106622 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:21.106846 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:21.107282 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:21.107287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:21.107290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:21.108036 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:21.108395 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:21.108851 (+ 456us) server_negotiation.cc:299] Negotiation successful
0812 02:35:21.109289 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":320,"threads_started":1}
I20260812 02:35:21.112897 6116 raft_consensus.cc:348] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68499eb3f8e4f419896900ead72d13d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37293 } }
I20260812 02:35:21.113310 6116 raft_consensus.cc:374] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:21.113417 6116 raft_consensus.cc:729] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68499eb3f8e4f419896900ead72d13d, State: Initialized, Role: FOLLOWER
I20260812 02:35:21.114037 6116 consensus_queue.cc:260] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68499eb3f8e4f419896900ead72d13d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37293 } }
I20260812 02:35:21.114301 6116 raft_consensus.cc:388] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:21.114437 6116 raft_consensus.cc:482] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:21.114589 6116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:21.114905 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:21.116907 6116 raft_consensus.cc:504] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68499eb3f8e4f419896900ead72d13d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37293 } }
I20260812 02:35:21.117486 6116 leader_election.cc:302] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d68499eb3f8e4f419896900ead72d13d; no voters:
I20260812 02:35:21.118131 6116 leader_election.cc:288] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:21.118274 6124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:21.120322 6116 sys_catalog.cc:564] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:21.121474 6124 raft_consensus.cc:686] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [term 1 LEADER]: Becoming Leader. State: Replica: d68499eb3f8e4f419896900ead72d13d, State: Running, Role: LEADER
I20260812 02:35:21.122166 6124 consensus_queue.cc:237] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68499eb3f8e4f419896900ead72d13d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37293 } }
I20260812 02:35:21.125367 6126 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d68499eb3f8e4f419896900ead72d13d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68499eb3f8e4f419896900ead72d13d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37293 } } }
I20260812 02:35:21.125795 6126 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:21.126335 6127 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d68499eb3f8e4f419896900ead72d13d. Latest consensus state: current_term: 1 leader_uuid: "d68499eb3f8e4f419896900ead72d13d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68499eb3f8e4f419896900ead72d13d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37293 } } }
I20260812 02:35:21.126614 6127 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:21.132027 6134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:21.137934 6134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:21.149339 6134 catalog_manager.cc:1347] Generated new cluster ID: 57d768c352f2408998c9c94f04c3226b
I20260812 02:35:21.149453 6134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:21.150243 6145 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:35:21.151288 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:21.123648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:40059 (local address 127.0.0.1:37293)
0812 02:35:21.123839 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:21.123845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:21.130694 (+ 6849us) server_negotiation.cc:407] Connection header received
0812 02:35:21.131965 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:21.131971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:21.132070 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:21.132196 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:21.134316 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:21.135736 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:21.137858 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:21.138634 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:21.138782 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:21.142454 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:21.142462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:21.142468 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:21.142520 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:21.146527 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:21.147640 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:21.147647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:21.147651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:21.147863 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:21.148441 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:21.148447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:21.148450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:21.148898 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:21.149197 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:21.150705 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0812 02:35:21.150917 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:35:21.171041 6134 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:21.172421 6134 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:21.183607 6134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d68499eb3f8e4f419896900ead72d13d: Generated new TSK 0
I20260812 02:35:21.184458 6134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:21.192873 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6039
I20260812 02:35:21.194077 6109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:21.287436 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 6039
W20260812 02:35:21.306972 6021 connection.cc:535] client connection to 127.0.0.1:37293 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37293): Connection reset by peer (error 104)
2026-08-12T02:35:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:21 dist-test-slave-vq2q krb5kdc[6152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:21 dist-test-slave-vq2q krb5kdc[6152](info): set up 2 sockets
Aug 12 02:35:21 dist-test-slave-vq2q krb5kdc[6152](info): commencing operation
krb5kdc: starting...
W20260812 02:35:22.451033 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.002s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:22 dist-test-slave-vq2q krb5kdc[6152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:22.870749 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:23.235262 6167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:23.235584 6167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:23.235708 6167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:23.242997 6167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:23.243141 6167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:23.243220 6167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:23.243300 6167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:23.243360 6167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:23.253518 6167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:23.254940 6167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:23.256297 6167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:23.268803 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:23.268867 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:23.268973 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:23.270925 6167 server_base.cc:1034] running on GCE node
I20260812 02:35:23.271752 6167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:23.273383 6167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:23.274590 6167 hybrid_clock.cc:648] HybridClock initialized: now 1786502123274537 us; error 56 us; skew 500 ppm
I20260812 02:35:23.274935 6167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:23 dist-test-slave-vq2q krb5kdc[6152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:23.281865 6167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:23.284037 6167 webserver.cc:460] Webserver started at http://127.0.0.1:37173/ using document root <none> and password file <none>
I20260812 02:35:23.284973 6167 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:23.285101 6167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:23.285486 6167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:23.289383 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "af1a419cb31f494d8fafc09611c17da2"
format_stamp: "Formatted at 2026-08-12 02:35:23 on dist-test-slave-vq2q"
server_key: "769bf0a0240f630c09d70688f5029c4b"
server_key_iv: "7e4314d5fe4ef9452c1961ecf23543f4"
server_key_version: "encryptionkey@0"
I20260812 02:35:23.290335 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "af1a419cb31f494d8fafc09611c17da2"
format_stamp: "Formatted at 2026-08-12 02:35:23 on dist-test-slave-vq2q"
server_key: "769bf0a0240f630c09d70688f5029c4b"
server_key_iv: "7e4314d5fe4ef9452c1961ecf23543f4"
server_key_version: "encryptionkey@0"
I20260812 02:35:23.298131 6167 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 02:35:23.303884 6183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:23.307157 6167 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260812 02:35:23.307464 6167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "af1a419cb31f494d8fafc09611c17da2"
format_stamp: "Formatted at 2026-08-12 02:35:23 on dist-test-slave-vq2q"
server_key: "769bf0a0240f630c09d70688f5029c4b"
server_key_iv: "7e4314d5fe4ef9452c1961ecf23543f4"
server_key_version: "encryptionkey@0"
I20260812 02:35:23.307760 6167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:23.330809 6167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:23.335995 6167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:23.336335 6167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:23.358948 6167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45591
I20260812 02:35:23.358976 6246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45591 every 8 connection(s)
I20260812 02:35:23.360790 6167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:23.364758 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 6167
I20260812 02:35:23.364961 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:23.365142 6247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:23.365618 12323 external_mini_cluster.cc:1428] Setting key 5cb1da8a0e25492623fd2ca2df28b661
I20260812 02:35:23.379746 6247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2: Bootstrap starting.
I20260812 02:35:23.382827 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:23.368120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:41659 (local address 127.0.0.1:45591)
0812 02:35:23.369161 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:23.369200 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:23.369278 (+ 78us) server_negotiation.cc:407] Connection header received
0812 02:35:23.371947 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:23.371978 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:23.372531 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:23.372993 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:23.373692 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:23.375552 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:23.377157 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:23.378189 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:23.378406 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:23.379831 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:23.379843 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:23.379860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:23.380608 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:23.380723 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:23.380758 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:23.380925 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:23.381298 (+ 373us) server_negotiation.cc:299] Negotiation successful
0812 02:35:23.381501 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":387,"threads_started":1}
I20260812 02:35:23.383821 6247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:23.385056 6247 log.cc:824] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:23.388176 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:23.389132 6247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2: No bootstrap required, opened a new log
I20260812 02:35:23.399314 6247 raft_consensus.cc:348] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1a419cb31f494d8fafc09611c17da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } }
I20260812 02:35:23.399991 6247 raft_consensus.cc:374] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:23.400123 6247 raft_consensus.cc:729] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1a419cb31f494d8fafc09611c17da2, State: Initialized, Role: FOLLOWER
I20260812 02:35:23.400485 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:23.392683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:51799 (local address 127.0.0.1:45591)
0812 02:35:23.392975 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:23.392982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:23.393395 (+ 413us) server_negotiation.cc:407] Connection header received
0812 02:35:23.393566 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:23.393572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:23.393677 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:23.393835 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:23.394466 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:23.395766 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:23.397301 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:23.398055 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:23.398246 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:23.399482 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:23.399488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:23.399493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:23.399523 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:23.399602 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:23.399628 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:23.399771 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:23.400057 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 02:35:23.400147 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260812 02:35:23.401022 6247 consensus_queue.cc:260] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1a419cb31f494d8fafc09611c17da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } }
I20260812 02:35:23.401391 6247 raft_consensus.cc:388] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:23.401562 6247 raft_consensus.cc:482] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:23.401768 6247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:23.404711 6247 raft_consensus.cc:504] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1a419cb31f494d8fafc09611c17da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } }
I20260812 02:35:23.405438 6247 leader_election.cc:302] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af1a419cb31f494d8fafc09611c17da2; no voters:
I20260812 02:35:23.406275 6247 leader_election.cc:288] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:23.406471 6257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:23.407732 6257 raft_consensus.cc:686] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [term 1 LEADER]: Becoming Leader. State: Replica: af1a419cb31f494d8fafc09611c17da2, State: Running, Role: LEADER
I20260812 02:35:23.408497 6257 consensus_queue.cc:237] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1a419cb31f494d8fafc09611c17da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } }
I20260812 02:35:23.409273 6247 sys_catalog.cc:564] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:23.415464 6260 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af1a419cb31f494d8fafc09611c17da2. Latest consensus state: current_term: 1 leader_uuid: "af1a419cb31f494d8fafc09611c17da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1a419cb31f494d8fafc09611c17da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } } }
I20260812 02:35:23.415509 6258 sys_catalog.cc:455] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af1a419cb31f494d8fafc09611c17da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1a419cb31f494d8fafc09611c17da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45591 } } }
I20260812 02:35:23.415963 6260 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:23.416010 6258 sys_catalog.cc:458] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:23.422840 6273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:23.428270 6273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:23.439836 6273 catalog_manager.cc:1347] Generated new cluster ID: 07c62f12a98e4aa9888752f34e1dd8dc
I20260812 02:35:23.439986 6273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:23.467257 6273 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:23.469056 6273 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:23.481823 6273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af1a419cb31f494d8fafc09611c17da2: Generated new TSK 0
I20260812 02:35:23.482970 6273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:23.501003 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6167
W20260812 02:35:23.503093 6242 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.26:41659
W20260812 02:35:23.504020 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6167: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:23.504196 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 6167
W20260812 02:35:23.529518 6150 connection.cc:535] client connection to 127.0.0.1:45591 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45591): Connection reset by peer (error 104)
2026-08-12T02:35:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:23 dist-test-slave-vq2q krb5kdc[6283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:23 dist-test-slave-vq2q krb5kdc[6283](info): set up 2 sockets
Aug 12 02:35:23 dist-test-slave-vq2q krb5kdc[6283](info): commencing operation
krb5kdc: starting...
W20260812 02:35:24.716447 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.081s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:24 dist-test-slave-vq2q krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:35:25.145402 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:25.553828 6300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:25.554158 6300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:25.554261 6300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:25.561961 6300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:25.562098 6300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:25.562145 6300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:25.562211 6300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:25.562289 6300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:25.572899 6300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:25.574389 6300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:25.575812 6300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:25.590396 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:25.590508 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:25.590462 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:25.591923 6300 server_base.cc:1034] running on GCE node
I20260812 02:35:25.592998 6300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:25.595034 6300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:25.596315 6300 hybrid_clock.cc:648] HybridClock initialized: now 1786502125596249 us; error 50 us; skew 500 ppm
I20260812 02:35:25.596819 6300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:25 dist-test-slave-vq2q krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:25.600875 6300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:35:25.602926 6300 webserver.cc:460] Webserver started at http://127.0.0.1:35267/ using document root <none> and password file <none>
I20260812 02:35:25.603852 6300 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:25.603997 6300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:25.604369 6300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:25.608165 6300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "146629e3cd06441ea1a55afc76eceaab"
format_stamp: "Formatted at 2026-08-12 02:35:25 on dist-test-slave-vq2q"
server_key: "c7776ecdd6b8ede5d22433057dd5b4bc"
server_key_iv: "54a002176178f442bc9abce5cb0759ef"
server_key_version: "encryptionkey@0"
I20260812 02:35:25.609172 6300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "146629e3cd06441ea1a55afc76eceaab"
format_stamp: "Formatted at 2026-08-12 02:35:25 on dist-test-slave-vq2q"
server_key: "c7776ecdd6b8ede5d22433057dd5b4bc"
server_key_iv: "54a002176178f442bc9abce5cb0759ef"
server_key_version: "encryptionkey@0"
I20260812 02:35:25.616315 6300 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 02:35:25.621269 6317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:25.623250 6300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:35:25.623512 6300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "146629e3cd06441ea1a55afc76eceaab"
format_stamp: "Formatted at 2026-08-12 02:35:25 on dist-test-slave-vq2q"
server_key: "c7776ecdd6b8ede5d22433057dd5b4bc"
server_key_iv: "54a002176178f442bc9abce5cb0759ef"
server_key_version: "encryptionkey@0"
I20260812 02:35:25.623780 6300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:25.631211 6300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:25.636773 6300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:25.637143 6300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:25.659560 6300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38027
I20260812 02:35:25.659617 6379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38027 every 8 connection(s)
I20260812 02:35:25.661514 6300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:25.666188 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:25.669056 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 6300
I20260812 02:35:25.669314 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:25.669878 12323 external_mini_cluster.cc:1428] Setting key ed5d44e7fc92c7cff80e192f57ff9e96
I20260812 02:35:25.680364 6380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab: Bootstrap starting.
I20260812 02:35:25.680469 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:25.672507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:36779 (local address 127.0.0.1:38027)
0812 02:35:25.673639 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:25.673657 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:25.673722 (+ 65us) server_negotiation.cc:407] Connection header received
0812 02:35:25.675889 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:25.675923 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:25.676471 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:25.676956 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:25.678342 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:25.678351 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:25.678369 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:25.678388 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:25.678467 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:25.678491 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:25.678732 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:25.678856 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 02:35:25.679134 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":420,"threads_started":1}
I20260812 02:35:25.684281 6380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:25.685588 6380 log.cc:824] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:25.685860 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:25.688623 6380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab: No bootstrap required, opened a new log
I20260812 02:35:25.694765 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:25.691601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:45019 (local address 127.0.0.1:38027)
0812 02:35:25.691891 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:25.691898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:25.692256 (+ 358us) server_negotiation.cc:407] Connection header received
0812 02:35:25.692483 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:25.692489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:25.692558 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:25.692672 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:25.693901 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:25.693906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:25.693910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:25.693934 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:25.693987 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:25.694055 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:25.694202 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:25.694325 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 02:35:25.694416 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260812 02:35:25.697933 6380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146629e3cd06441ea1a55afc76eceaab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38027 } }
I20260812 02:35:25.698352 6380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:25.698450 6380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146629e3cd06441ea1a55afc76eceaab, State: Initialized, Role: FOLLOWER
I20260812 02:35:25.699041 6380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146629e3cd06441ea1a55afc76eceaab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38027 } }
I20260812 02:35:25.699268 6380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:25.699415 6380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:25.699568 6380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:25.702036 6380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146629e3cd06441ea1a55afc76eceaab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38027 } }
I20260812 02:35:25.702620 6380 leader_election.cc:302] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 146629e3cd06441ea1a55afc76eceaab; no voters:
I20260812 02:35:25.703434 6380 leader_election.cc:288] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:25.703691 6395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:25.704524 6395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [term 1 LEADER]: Becoming Leader. State: Replica: 146629e3cd06441ea1a55afc76eceaab, State: Running, Role: LEADER
I20260812 02:35:25.705323 6395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146629e3cd06441ea1a55afc76eceaab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38027 } }
I20260812 02:35:25.705929 6380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:25.709102 6398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 146629e3cd06441ea1a55afc76eceaab. Latest consensus state: current_term: 1 leader_uuid: "146629e3cd06441ea1a55afc76eceaab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146629e3cd06441ea1a55afc76eceaab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38027 } } }
I20260812 02:35:25.709692 6398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:25.710281 6397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "146629e3cd06441ea1a55afc76eceaab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146629e3cd06441ea1a55afc76eceaab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38027 } } }
I20260812 02:35:25.710701 6397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:25.717908 6406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:25.723826 6406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:25.734965 6406 catalog_manager.cc:1347] Generated new cluster ID: 3fe3a42dc507470ba598e30b2e5ae727
I20260812 02:35:25.735140 6406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:25.755076 6406 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:25.755721 6406 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:25.765915 6406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 146629e3cd06441ea1a55afc76eceaab: Generated new TSK 0
I20260812 02:35:25.767096 6406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:25.777839 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6300
W20260812 02:35:25.787369 6374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.26:36779
W20260812 02:35:25.788435 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:25.788650 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 6300
2026-08-12T02:35:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:25 dist-test-slave-vq2q krb5kdc[6419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:25 dist-test-slave-vq2q krb5kdc[6419](info): set up 2 sockets
Aug 12 02:35:25 dist-test-slave-vq2q krb5kdc[6419](info): commencing operation
krb5kdc: starting...
W20260812 02:35:27.021597 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.007s sys 0.076s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:27 dist-test-slave-vq2q krb5kdc[6419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:35:27.476836 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43431
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:27.959086 6433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:27.959507 6433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:27.959641 6433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:27.967921 6433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:27.968096 6433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:27.968158 6433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:27.968261 6433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:27.968322 6433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:27.979969 6433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:43431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:27.981822 6433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:27.983554 6433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:27.998273 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:27.998353 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:27.998273 6445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:27.999874 6433 server_base.cc:1034] running on GCE node
I20260812 02:35:28.001070 6433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:28.003230 6433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:28.004491 6433 hybrid_clock.cc:648] HybridClock initialized: now 1786502128004437 us; error 51 us; skew 500 ppm
I20260812 02:35:28.004976 6433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:28 dist-test-slave-vq2q krb5kdc[6419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502128, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:28.009158 6433 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:35:28.011417 6433 webserver.cc:460] Webserver started at http://127.12.8.254:41889/ using document root <none> and password file <none>
I20260812 02:35:28.012403 6433 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:28.012575 6433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:28.013056 6433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:28.017213 6433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "dae18ec2f3434414a53ab73820ed50ed"
format_stamp: "Formatted at 2026-08-12 02:35:28 on dist-test-slave-vq2q"
server_key: "bdad24e1632357efe962ffbab89dd404"
server_key_iv: "5772337db562ca18ae8939a14f7cb4ac"
server_key_version: "encryptionkey@0"
I20260812 02:35:28.018312 6433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "dae18ec2f3434414a53ab73820ed50ed"
format_stamp: "Formatted at 2026-08-12 02:35:28 on dist-test-slave-vq2q"
server_key: "bdad24e1632357efe962ffbab89dd404"
server_key_iv: "5772337db562ca18ae8939a14f7cb4ac"
server_key_version: "encryptionkey@0"
I20260812 02:35:28.025403 6433 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 02:35:28.030470 6452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:28.032636 6433 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:35:28.032918 6433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "dae18ec2f3434414a53ab73820ed50ed"
format_stamp: "Formatted at 2026-08-12 02:35:28 on dist-test-slave-vq2q"
server_key: "bdad24e1632357efe962ffbab89dd404"
server_key_iv: "5772337db562ca18ae8939a14f7cb4ac"
server_key_version: "encryptionkey@0"
I20260812 02:35:28.033219 6433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:28.066852 6433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:28.072889 6433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:28.073356 6433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:28.097606 6433 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:43431
I20260812 02:35:28.097647 6510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:43431 every 8 connection(s)
I20260812 02:35:28.099556 6433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:28.102748 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 6433
I20260812 02:35:28.102983 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:28.103612 12323 external_mini_cluster.cc:1428] Setting key 97870ecb49097dc5c348d59092b7fe2e
I20260812 02:35:28.104480 6511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:28.120441 6511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed: Bootstrap starting.
Aug 12 02:35:28 dist-test-slave-vq2q krb5kdc[6419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:28.124516 6511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:28.126016 6511 log.cc:824] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:28.129882 6511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed: No bootstrap required, opened a new log
I20260812 02:35:28.133466 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:28.106209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56111 (local address 127.12.8.254:43431)
0812 02:35:28.107271 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:28.107286 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:28.107349 (+ 63us) server_negotiation.cc:407] Connection header received
0812 02:35:28.110073 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:28.110114 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:28.110703 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:28.111189 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:28.113304 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:28.114800 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:28.116827 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:28.117838 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:28.118018 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:28.123705 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:28.123718 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:28.123741 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:28.123813 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:28.127766 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:28.129024 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:28.129032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:28.129040 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:28.129317 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:28.129883 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:28.129887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:28.129889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:28.130595 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:28.130910 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:28.131499 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 02:35:28.131932 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":488,"threads_started":1}
I20260812 02:35:28.138669 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:28.139731 6511 raft_consensus.cc:348] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae18ec2f3434414a53ab73820ed50ed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43431 } }
I20260812 02:35:28.140224 6511 raft_consensus.cc:374] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:28.140321 6511 raft_consensus.cc:729] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae18ec2f3434414a53ab73820ed50ed, State: Initialized, Role: FOLLOWER
I20260812 02:35:28.140952 6511 consensus_queue.cc:260] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae18ec2f3434414a53ab73820ed50ed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43431 } }
I20260812 02:35:28.141211 6511 raft_consensus.cc:388] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:28.141356 6511 raft_consensus.cc:482] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:28.141556 6511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:28.144411 6511 raft_consensus.cc:504] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae18ec2f3434414a53ab73820ed50ed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43431 } }
I20260812 02:35:28.145059 6511 leader_election.cc:302] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dae18ec2f3434414a53ab73820ed50ed; no voters:
I20260812 02:35:28.145853 6511 leader_election.cc:288] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:28.145982 6523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:28.148546 6511 sys_catalog.cc:564] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:28.152613 6523 raft_consensus.cc:686] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [term 1 LEADER]: Becoming Leader. State: Replica: dae18ec2f3434414a53ab73820ed50ed, State: Running, Role: LEADER
I20260812 02:35:28.153720 6523 consensus_queue.cc:237] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae18ec2f3434414a53ab73820ed50ed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43431 } }
I20260812 02:35:28.157887 6524 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dae18ec2f3434414a53ab73820ed50ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae18ec2f3434414a53ab73820ed50ed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43431 } } }
I20260812 02:35:28.157969 6525 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader dae18ec2f3434414a53ab73820ed50ed. Latest consensus state: current_term: 1 leader_uuid: "dae18ec2f3434414a53ab73820ed50ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae18ec2f3434414a53ab73820ed50ed" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 43431 } } }
I20260812 02:35:28.158504 6524 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:28.158504 6525 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:28.163473 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:28.164897 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:28.143867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55601 (local address 127.12.8.254:43431)
0812 02:35:28.144054 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:28.144059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:28.144688 (+ 629us) server_negotiation.cc:407] Connection header received
0812 02:35:28.144890 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:28.144896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:28.144996 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:28.145139 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:28.147187 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:28.148063 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:28.149660 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:28.150313 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:28.150473 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:28.154392 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:28.154403 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:28.154408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:28.154471 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:28.160005 (+ 5534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:28.161766 (+ 1761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:28.161775 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:28.161779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:28.162015 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:28.162682 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:28.162689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:28.162693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:28.163520 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:28.163844 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:28.164295 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 02:35:28.164530 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:35:28.169011 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:28.178743 6536 catalog_manager.cc:1347] Generated new cluster ID: c6f274582df842659825ce758ecc0c34
I20260812 02:35:28.178897 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:28.204875 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:28.206576 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:28.213605 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dae18ec2f3434414a53ab73820ed50ed: Generated new TSK 0
I20260812 02:35:28.215190 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:28.228909 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6433
I20260812 02:35:28.231218 6506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:28.331631 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 6433
W20260812 02:35:28.357388 6417 connection.cc:535] client connection to 127.12.8.254:43431 recv error: Network error: failed to read from TLS socket (remote: 127.12.8.254:43431): Connection reset by peer (error 104)
2026-08-12T02:35:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2542 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:28 dist-test-slave-vq2q krb5kdc[6548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:28 dist-test-slave-vq2q krb5kdc[6548](info): set up 2 sockets
Aug 12 02:35:28 dist-test-slave-vq2q krb5kdc[6548](info): commencing operation
krb5kdc: starting...
W20260812 02:35:29.556288 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:29 dist-test-slave-vq2q krb5kdc[6548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:35:30.014714 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:35679
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:33009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:35679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:30.408071 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:30.408442 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:30.408542 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:30.416232 6562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:30.416373 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:30.416422 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:30.416477 6562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:30.416530 6562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:30.426932 6562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:33009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:35679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:35679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:30.428334 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:30.429679 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:30.446484 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:30.446722 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:30.446818 6562 server_base.cc:1034] running on GCE node
W20260812 02:35:30.445779 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:30.448166 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:30.449864 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:30.451107 6562 hybrid_clock.cc:648] HybridClock initialized: now 1786502130451041 us; error 58 us; skew 500 ppm
I20260812 02:35:30.451507 6562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:30 dist-test-slave-vq2q krb5kdc[6548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:30.455154 6562 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:35:30.456907 6562 webserver.cc:460] Webserver started at http://127.12.8.254:38751/ using document root <none> and password file <none>
I20260812 02:35:30.457682 6562 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:30.457821 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:30.458137 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:30.461409 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "9d51001adc9e48ddac1da9d8a5b9f655"
format_stamp: "Formatted at 2026-08-12 02:35:30 on dist-test-slave-vq2q"
server_key: "edf630620f3e199786f2afdf74eb3952"
server_key_iv: "71a23cc5101df58a9c88cbc15858ba2c"
server_key_version: "encryptionkey@0"
I20260812 02:35:30.462344 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "9d51001adc9e48ddac1da9d8a5b9f655"
format_stamp: "Formatted at 2026-08-12 02:35:30 on dist-test-slave-vq2q"
server_key: "edf630620f3e199786f2afdf74eb3952"
server_key_iv: "71a23cc5101df58a9c88cbc15858ba2c"
server_key_version: "encryptionkey@0"
I20260812 02:35:30.468802 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 02:35:30.473073 6581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:30.474910 6562 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 02:35:30.475126 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "9d51001adc9e48ddac1da9d8a5b9f655"
format_stamp: "Formatted at 2026-08-12 02:35:30 on dist-test-slave-vq2q"
server_key: "edf630620f3e199786f2afdf74eb3952"
server_key_iv: "71a23cc5101df58a9c88cbc15858ba2c"
server_key_version: "encryptionkey@0"
I20260812 02:35:30.475338 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:30.488830 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:30.493848 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:30.494164 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:30.515962 6562 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:35679
I20260812 02:35:30.516028 6636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:35679 every 8 connection(s)
I20260812 02:35:30.517549 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:30.522313 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 6562
I20260812 02:35:30.522513 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:30.522868 6637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:30.522989 12323 external_mini_cluster.cc:1428] Setting key c7dc1a48251433bdacd885f55ec11378
I20260812 02:35:30.538632 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:30.525149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35661 (local address 127.12.8.254:35679)
0812 02:35:30.526238 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:30.526256 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:30.526336 (+ 80us) server_negotiation.cc:407] Connection header received
0812 02:35:30.528823 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:30.528850 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:30.529418 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:30.529879 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:30.530523 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:30.531956 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:30.533369 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:30.534198 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:30.534410 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:30.535965 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:30.535974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:30.535990 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:30.536665 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:30.536772 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:30.536808 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:30.536980 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:30.537339 (+ 359us) server_negotiation.cc:299] Negotiation successful
0812 02:35:30.537524 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":482,"threads_started":1}
I20260812 02:35:30.540838 6637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655: Bootstrap starting.
I20260812 02:35:30.543977 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:30.545202 6637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:30.546502 6637 log.cc:824] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:30.550573 6637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655: No bootstrap required, opened a new log
I20260812 02:35:30.557389 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:30.549337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60205 (local address 127.12.8.254:35679)
0812 02:35:30.549566 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:30.549573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:30.549828 (+ 255us) server_negotiation.cc:407] Connection header received
0812 02:35:30.550035 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:30.550041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:30.550142 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:30.550298 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:30.550878 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:30.552090 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:30.553637 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:30.554751 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:30.554956 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:30.556345 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:30.556353 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:30.556358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:30.556390 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:30.556454 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:30.556477 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:30.556647 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:30.556845 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 02:35:30.556930 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:35:30.562242 6637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51001adc9e48ddac1da9d8a5b9f655" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35679 } }
I20260812 02:35:30.562824 6637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:30.562914 6637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d51001adc9e48ddac1da9d8a5b9f655, State: Initialized, Role: FOLLOWER
I20260812 02:35:30.563661 6637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51001adc9e48ddac1da9d8a5b9f655" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35679 } }
I20260812 02:35:30.564019 6637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:30.564222 6637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:30.564447 6637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:30.567423 6637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51001adc9e48ddac1da9d8a5b9f655" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35679 } }
I20260812 02:35:30.568217 6637 leader_election.cc:302] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d51001adc9e48ddac1da9d8a5b9f655; no voters:
I20260812 02:35:30.569118 6637 leader_election.cc:288] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:30.569284 6650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:30.570200 6650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [term 1 LEADER]: Becoming Leader. State: Replica: 9d51001adc9e48ddac1da9d8a5b9f655, State: Running, Role: LEADER
I20260812 02:35:30.570863 6650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51001adc9e48ddac1da9d8a5b9f655" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35679 } }
I20260812 02:35:30.572124 6637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:30.574472 6651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d51001adc9e48ddac1da9d8a5b9f655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51001adc9e48ddac1da9d8a5b9f655" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35679 } } }
I20260812 02:35:30.574965 6651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:30.578970 6652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d51001adc9e48ddac1da9d8a5b9f655. Latest consensus state: current_term: 1 leader_uuid: "9d51001adc9e48ddac1da9d8a5b9f655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51001adc9e48ddac1da9d8a5b9f655" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 35679 } } }
I20260812 02:35:30.579360 6652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:30.580197 6660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:30.586333 6660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:30.599685 6660 catalog_manager.cc:1347] Generated new cluster ID: f6951c605c494204914f21acb3164bb1
I20260812 02:35:30.599836 6660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:30.616583 6660 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:30.618378 6660 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:30.630790 6660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d51001adc9e48ddac1da9d8a5b9f655: Generated new TSK 0
I20260812 02:35:30.632077 6660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:30.650866 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6562
W20260812 02:35:30.652832 6632 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35661
W20260812 02:35:30.653926 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6562: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:30.654148 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 6562
W20260812 02:35:30.677367 6546 connection.cc:535] client connection to 127.12.8.254:35679 recv error: Network error: failed to read from TLS socket (remote: 127.12.8.254:35679): Connection reset by peer (error 104)
2026-08-12T02:35:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:30 dist-test-slave-vq2q krb5kdc[6673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:30 dist-test-slave-vq2q krb5kdc[6673](info): set up 2 sockets
Aug 12 02:35:30 dist-test-slave-vq2q krb5kdc[6673](info): commencing operation
krb5kdc: starting...
W20260812 02:35:31.885597 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.001s sys 0.084s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:32 dist-test-slave-vq2q krb5kdc[6673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:35:32.324828 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46423
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:46423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:32.746794 6690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:32.747112 6690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:32.747208 6690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:32.754665 6690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:32.754809 6690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:32.754858 6690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:32.754935 6690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:32.754985 6690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:32.765784 6690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:46423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:46423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:32.767228 6690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:32.768611 6690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:32.782594 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:32.782649 6696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:32.782691 6695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:32.784222 6690 server_base.cc:1034] running on GCE node
I20260812 02:35:32.785609 6690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:32.787712 6690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:32.789003 6690 hybrid_clock.cc:648] HybridClock initialized: now 1786502132788928 us; error 63 us; skew 500 ppm
I20260812 02:35:32.789580 6690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:32 dist-test-slave-vq2q krb5kdc[6673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502132, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:32.794011 6690 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:35:32.796074 6690 webserver.cc:460] Webserver started at http://127.12.8.254:39475/ using document root <none> and password file <none>
I20260812 02:35:32.797034 6690 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:32.797210 6690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:32.797650 6690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:32.802248 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "8a2a9a3d165843a18b956ef60d78eeea"
format_stamp: "Formatted at 2026-08-12 02:35:32 on dist-test-slave-vq2q"
server_key: "ebbc9a52e37f17ff18e0a7a21a438710"
server_key_iv: "99d3e67fb149d9f0ddbb7d58a0b14c8b"
server_key_version: "encryptionkey@0"
I20260812 02:35:32.803566 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "8a2a9a3d165843a18b956ef60d78eeea"
format_stamp: "Formatted at 2026-08-12 02:35:32 on dist-test-slave-vq2q"
server_key: "ebbc9a52e37f17ff18e0a7a21a438710"
server_key_iv: "99d3e67fb149d9f0ddbb7d58a0b14c8b"
server_key_version: "encryptionkey@0"
I20260812 02:35:32.811096 6690 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 02:35:32.816373 6706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:32.818581 6690 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:35:32.818888 6690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "8a2a9a3d165843a18b956ef60d78eeea"
format_stamp: "Formatted at 2026-08-12 02:35:32 on dist-test-slave-vq2q"
server_key: "ebbc9a52e37f17ff18e0a7a21a438710"
server_key_iv: "99d3e67fb149d9f0ddbb7d58a0b14c8b"
server_key_version: "encryptionkey@0"
I20260812 02:35:32.819166 6690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:32.828101 6690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:32.834378 6690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:32.834815 6690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:32.857692 6690 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:46423
I20260812 02:35:32.857735 6769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:46423 every 8 connection(s)
I20260812 02:35:32.859664 6690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:32.860399 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 6690
I20260812 02:35:32.860580 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:32.861198 12323 external_mini_cluster.cc:1428] Setting key c196b078c9553dd532ca8d883069ad3a
I20260812 02:35:32.866297 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:32.870630 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:32.863428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47123 (local address 127.12.8.254:46423)
0812 02:35:32.864388 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:32.864406 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:32.864573 (+ 167us) server_negotiation.cc:407] Connection header received
0812 02:35:32.866689 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:32.866716 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:32.867133 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:32.867543 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:32.868723 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:32.868730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:32.868744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:32.868767 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:32.868854 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:32.868878 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:32.869116 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:32.869309 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 02:35:32.869513 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":421,"threads_started":1}
I20260812 02:35:32.876300 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:32.884734 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:32.881598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50755 (local address 127.12.8.254:46423)
0812 02:35:32.881783 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:32.881789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:32.882376 (+ 587us) server_negotiation.cc:407] Connection header received
0812 02:35:32.882567 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:32.882573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:32.882636 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:32.882729 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:32.883990 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:32.884004 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:32.884009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:32.884031 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:32.884078 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:32.884102 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:32.884218 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:32.884341 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 02:35:32.884418 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:35:32.884965 6770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea: Bootstrap starting.
I20260812 02:35:32.891512 6770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:32.893410 6770 log.cc:824] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:32.897883 6770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea: No bootstrap required, opened a new log
I20260812 02:35:32.908255 6770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a9a3d165843a18b956ef60d78eeea" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46423 } }
I20260812 02:35:32.908856 6770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:32.908972 6770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2a9a3d165843a18b956ef60d78eeea, State: Initialized, Role: FOLLOWER
I20260812 02:35:32.910365 6770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a9a3d165843a18b956ef60d78eeea" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46423 } }
I20260812 02:35:32.910806 6770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:32.910976 6770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:32.911186 6770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:32.914152 6770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a9a3d165843a18b956ef60d78eeea" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46423 } }
I20260812 02:35:32.914971 6770 leader_election.cc:302] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a2a9a3d165843a18b956ef60d78eeea; no voters:
I20260812 02:35:32.915907 6770 leader_election.cc:288] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:32.916237 6783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:32.917598 6783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [term 1 LEADER]: Becoming Leader. State: Replica: 8a2a9a3d165843a18b956ef60d78eeea, State: Running, Role: LEADER
I20260812 02:35:32.918521 6783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a9a3d165843a18b956ef60d78eeea" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46423 } }
I20260812 02:35:32.919642 6770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:32.922077 6785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a2a9a3d165843a18b956ef60d78eeea. Latest consensus state: current_term: 1 leader_uuid: "8a2a9a3d165843a18b956ef60d78eeea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a9a3d165843a18b956ef60d78eeea" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46423 } } }
I20260812 02:35:32.922551 6785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:32.923468 6784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a2a9a3d165843a18b956ef60d78eeea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a9a3d165843a18b956ef60d78eeea" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 46423 } } }
I20260812 02:35:32.923815 6784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:32.924929 6790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:32.930661 6790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:32.940986 6790 catalog_manager.cc:1347] Generated new cluster ID: 5a6a1b2b40874e49a3e524bed4b977b4
I20260812 02:35:32.941143 6790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:32.973080 6790 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:32.973634 6790 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:32.980705 6790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a2a9a3d165843a18b956ef60d78eeea: Generated new TSK 0
I20260812 02:35:32.981703 6790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:32.997838 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6690
W20260812 02:35:32.999274 6765 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47123
W20260812 02:35:33.000304 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6690: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:33.000465 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 6690
2026-08-12T02:35:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:33 dist-test-slave-vq2q krb5kdc[6807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:33 dist-test-slave-vq2q krb5kdc[6807](info): set up 2 sockets
Aug 12 02:35:33 dist-test-slave-vq2q krb5kdc[6807](info): commencing operation
krb5kdc: starting...
W20260812 02:35:34.194427 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.004s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:34 dist-test-slave-vq2q krb5kdc[6807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:35:34.616668 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33997
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:35.088804 6825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:35.089345 6825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:35.089512 6825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:35.098605 6825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:35.098824 6825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:35.098927 6825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:35.099040 6825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:35.099118 6825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:35.111750 6825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:35.113693 6825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:35.115739 6825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:35.131973 6830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:35.131958 6831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:35.131922 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:35.132881 6825 server_base.cc:1034] running on GCE node
I20260812 02:35:35.134280 6825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:35.136391 6825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:35.137686 6825 hybrid_clock.cc:648] HybridClock initialized: now 1786502135137620 us; error 58 us; skew 500 ppm
I20260812 02:35:35.138216 6825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:35 dist-test-slave-vq2q krb5kdc[6807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502135, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:35.150498 6825 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:35:35.152765 6825 webserver.cc:460] Webserver started at http://127.12.8.254:36963/ using document root <none> and password file <none>
I20260812 02:35:35.153834 6825 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:35.153988 6825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:35.154422 6825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:35.158526 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "bf35f967424045acb674958bd7564dc9"
format_stamp: "Formatted at 2026-08-12 02:35:35 on dist-test-slave-vq2q"
server_key: "824fc66335defcdd90978db338d0a8c2"
server_key_iv: "ac41fe7ef7f425a3f25a04f0af892b61"
server_key_version: "encryptionkey@0"
I20260812 02:35:35.159544 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "bf35f967424045acb674958bd7564dc9"
format_stamp: "Formatted at 2026-08-12 02:35:35 on dist-test-slave-vq2q"
server_key: "824fc66335defcdd90978db338d0a8c2"
server_key_iv: "ac41fe7ef7f425a3f25a04f0af892b61"
server_key_version: "encryptionkey@0"
I20260812 02:35:35.166903 6825 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 02:35:35.172341 6844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:35.174549 6825 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 02:35:35.174834 6825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "bf35f967424045acb674958bd7564dc9"
format_stamp: "Formatted at 2026-08-12 02:35:35 on dist-test-slave-vq2q"
server_key: "824fc66335defcdd90978db338d0a8c2"
server_key_iv: "ac41fe7ef7f425a3f25a04f0af892b61"
server_key_version: "encryptionkey@0"
I20260812 02:35:35.175098 6825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:35.200294 6825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:35.206416 6825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:35.206820 6825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:35.230842 6825 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:33997
I20260812 02:35:35.230906 6910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:33997 every 8 connection(s)
I20260812 02:35:35.232679 6825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:35.237224 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:35.242527 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 6825
I20260812 02:35:35.242771 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:35.243371 12323 external_mini_cluster.cc:1428] Setting key a865ec491ff4d6f7babda79912fa82e8
I20260812 02:35:35.249938 6911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9: Bootstrap starting.
I20260812 02:35:35.254141 6911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:35.255889 6911 log.cc:824] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:35.260514 6911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9: No bootstrap required, opened a new log
Aug 12 02:35:35 dist-test-slave-vq2q krb5kdc[6807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:35.270784 6911 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf35f967424045acb674958bd7564dc9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33997 } }
I20260812 02:35:35.271335 6911 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:35.271453 6911 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf35f967424045acb674958bd7564dc9, State: Initialized, Role: FOLLOWER
I20260812 02:35:35.272192 6911 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf35f967424045acb674958bd7564dc9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33997 } }
I20260812 02:35:35.272495 6911 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:35.272662 6911 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:35.272856 6911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:35.274770 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:35.245998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36863 (local address 127.12.8.254:33997)
0812 02:35:35.247120 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:35.247147 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:35.247207 (+ 60us) server_negotiation.cc:407] Connection header received
0812 02:35:35.249423 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:35.249459 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:35.250041 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:35.250550 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:35.252571 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:35.254477 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:35.256707 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:35.258158 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:35.258371 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:35.264056 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:35.264073 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:35.264099 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:35.264162 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:35.268618 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:35.269968 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:35.269978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:35.269988 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:35.270225 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:35.270856 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:35.270862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:35.270865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:35.271689 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:35.272031 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:35.272677 (+ 646us) server_negotiation.cc:299] Negotiation successful
0812 02:35:35.273228 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":506,"threads_started":1}
I20260812 02:35:35.276038 6911 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf35f967424045acb674958bd7564dc9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33997 } }
I20260812 02:35:35.276634 6911 leader_election.cc:302] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf35f967424045acb674958bd7564dc9; no voters:
I20260812 02:35:35.277357 6911 leader_election.cc:288] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:35.277519 6918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:35.279193 6918 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [term 1 LEADER]: Becoming Leader. State: Replica: bf35f967424045acb674958bd7564dc9, State: Running, Role: LEADER
I20260812 02:35:35.279975 6911 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:35.279891 6918 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf35f967424045acb674958bd7564dc9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33997 } }
I20260812 02:35:35.280754 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:35.286285 6921 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf35f967424045acb674958bd7564dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf35f967424045acb674958bd7564dc9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33997 } } }
I20260812 02:35:35.286799 6921 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:35.287420 6922 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf35f967424045acb674958bd7564dc9. Latest consensus state: current_term: 1 leader_uuid: "bf35f967424045acb674958bd7564dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf35f967424045acb674958bd7564dc9" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 33997 } } }
I20260812 02:35:35.287725 6922 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:35.287868 6930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:35.297430 6930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:35.319543 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:35.291808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45087 (local address 127.12.8.254:33997)
0812 02:35:35.292035 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:35.292043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:35.297588 (+ 5545us) server_negotiation.cc:407] Connection header received
0812 02:35:35.297935 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:35.297941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:35.298044 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:35.298147 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:35.300649 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:35.302030 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:35.304257 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:35.304983 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:35.305233 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:35.309497 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:35.309507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:35.309513 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:35.309582 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:35.314755 (+ 5173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:35.316194 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:35.316203 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:35.316207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:35.316435 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:35.316990 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:35.316997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:35.317001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:35.317606 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:35.317901 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:35.318676 (+ 775us) server_negotiation.cc:299] Negotiation successful
0812 02:35:35.319016 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 02:35:35.324623 6930 catalog_manager.cc:1347] Generated new cluster ID: 3c38a34f6a1d4b818a7dc87b53470d9a
I20260812 02:35:35.324784 6930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:35.350670 6930 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:35.353092 6930 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:35.362739 6930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf35f967424045acb674958bd7564dc9: Generated new TSK 0
I20260812 02:35:35.363971 6930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:35.371309 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6825
I20260812 02:35:35.372925 6905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:35.481294 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 6825
2026-08-12T02:35:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:35 dist-test-slave-vq2q krb5kdc[6952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:35 dist-test-slave-vq2q krb5kdc[6952](info): set up 2 sockets
Aug 12 02:35:35 dist-test-slave-vq2q krb5kdc[6952](info): commencing operation
krb5kdc: starting...
W20260812 02:35:36.696622 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.004s sys 0.074s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:36 dist-test-slave-vq2q krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:35:37.138912 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40651
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:37.598275 6968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:37.598709 6968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:37.598855 6968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:37.607139 6968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:37.607322 6968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:37.607411 6968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:37.607513 6968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:37.607584 6968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:37.619334 6968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:37.621232 6968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:37.622929 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:37.637807 6973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:37.637738 6974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:37.637697 6977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:37.638690 6968 server_base.cc:1034] running on GCE node
I20260812 02:35:37.639978 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:37.642015 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:37.643396 6968 hybrid_clock.cc:648] HybridClock initialized: now 1786502137643310 us; error 57 us; skew 500 ppm
I20260812 02:35:37.643947 6968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:37 dist-test-slave-vq2q krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502137, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:37.653337 6968 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:35:37.655458 6968 webserver.cc:460] Webserver started at http://127.12.8.254:35239/ using document root <none> and password file <none>
I20260812 02:35:37.656281 6968 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:37.656415 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:37.656767 6968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:37.660272 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "61117919fc774a3c9ac3b3a1f980de00"
format_stamp: "Formatted at 2026-08-12 02:35:37 on dist-test-slave-vq2q"
server_key: "703602f5935bbc91a0f0d1000c0f0b01"
server_key_iv: "bc1047289a537433c572587369bd42d2"
server_key_version: "encryptionkey@0"
I20260812 02:35:37.661119 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "61117919fc774a3c9ac3b3a1f980de00"
format_stamp: "Formatted at 2026-08-12 02:35:37 on dist-test-slave-vq2q"
server_key: "703602f5935bbc91a0f0d1000c0f0b01"
server_key_iv: "bc1047289a537433c572587369bd42d2"
server_key_version: "encryptionkey@0"
I20260812 02:35:37.667639 6968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 02:35:37.672084 6985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:37.673882 6968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:35:37.674068 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "61117919fc774a3c9ac3b3a1f980de00"
format_stamp: "Formatted at 2026-08-12 02:35:37 on dist-test-slave-vq2q"
server_key: "703602f5935bbc91a0f0d1000c0f0b01"
server_key_iv: "bc1047289a537433c572587369bd42d2"
server_key_version: "encryptionkey@0"
I20260812 02:35:37.674276 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:37.686172 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:37.690780 6968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:37.691087 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:37.712802 6968 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:40651
I20260812 02:35:37.712893 7044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:40651 every 8 connection(s)
I20260812 02:35:37.714463 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:37.719702 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:37.725133 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 6968
I20260812 02:35:37.725445 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:37.726171 12323 external_mini_cluster.cc:1428] Setting key 5a1c28dfb97196bb8adafb2a2625212b
I20260812 02:35:37.737538 7046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00: Bootstrap starting.
I20260812 02:35:37.742238 7046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:37.743059 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:37.728868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57551 (local address 127.12.8.254:40651)
0812 02:35:37.730055 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:37.730072 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:37.730126 (+ 54us) server_negotiation.cc:407] Connection header received
0812 02:35:37.732369 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:37.732400 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:37.732950 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:37.733403 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:37.734290 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:37.735784 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:37.737869 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:37.738670 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:37.738821 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:37.740339 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:37.740348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:37.740365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:37.740999 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:37.741090 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:37.741118 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:37.741261 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:37.741507 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 02:35:37.741785 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":561,"threads_started":1}
I20260812 02:35:37.743636 7046 log.cc:824] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:37.747669 7046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00: No bootstrap required, opened a new log
I20260812 02:35:37.747951 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:37.756915 7046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61117919fc774a3c9ac3b3a1f980de00" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40651 } }
I20260812 02:35:37.757541 7046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:37.757668 7046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61117919fc774a3c9ac3b3a1f980de00, State: Initialized, Role: FOLLOWER
I20260812 02:35:37.758435 7046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61117919fc774a3c9ac3b3a1f980de00" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40651 } }
I20260812 02:35:37.758731 7046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:37.758893 7046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:37.759107 7046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:37.759353 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:37.752895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49893 (local address 127.12.8.254:40651)
0812 02:35:37.753198 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:37.753206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:37.753698 (+ 492us) server_negotiation.cc:407] Connection header received
0812 02:35:37.753903 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:37.753907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:37.753985 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:37.754081 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:37.754657 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:37.755431 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:37.756827 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:37.757321 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:37.757440 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:37.758553 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:37.758558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:37.758561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:37.758579 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:37.758623 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:37.758640 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:37.758723 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:37.758863 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 02:35:37.758964 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:35:37.762566 7046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61117919fc774a3c9ac3b3a1f980de00" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40651 } }
I20260812 02:35:37.763489 7046 leader_election.cc:302] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61117919fc774a3c9ac3b3a1f980de00; no voters:
I20260812 02:35:37.764300 7046 leader_election.cc:288] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:37.764469 7059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:37.765890 7059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [term 1 LEADER]: Becoming Leader. State: Replica: 61117919fc774a3c9ac3b3a1f980de00, State: Running, Role: LEADER
I20260812 02:35:37.766651 7059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61117919fc774a3c9ac3b3a1f980de00" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40651 } }
I20260812 02:35:37.767714 7046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:37.769966 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61117919fc774a3c9ac3b3a1f980de00. Latest consensus state: current_term: 1 leader_uuid: "61117919fc774a3c9ac3b3a1f980de00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61117919fc774a3c9ac3b3a1f980de00" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40651 } } }
I20260812 02:35:37.769750 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61117919fc774a3c9ac3b3a1f980de00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61117919fc774a3c9ac3b3a1f980de00" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40651 } } }
I20260812 02:35:37.770434 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:37.770455 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:37.771337 7066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:37.783020 7066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:37.796404 7066 catalog_manager.cc:1347] Generated new cluster ID: 1568dd533c2f448e80cf1a465d1eb3d1
I20260812 02:35:37.796636 7066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:37.831796 7066 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:37.833890 7066 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:37.849079 7066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61117919fc774a3c9ac3b3a1f980de00: Generated new TSK 0
I20260812 02:35:37.850453 7066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:37.863339 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6968
W20260812 02:35:37.865351 7038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57551
W20260812 02:35:37.866175 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 6968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:37.866349 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 6968
2026-08-12T02:35:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:38 dist-test-slave-vq2q krb5kdc[7089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:38 dist-test-slave-vq2q krb5kdc[7089](info): set up 2 sockets
Aug 12 02:35:38 dist-test-slave-vq2q krb5kdc[7089](info): commencing operation
krb5kdc: starting...
W20260812 02:35:39.078135 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.007s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:39 dist-test-slave-vq2q krb5kdc[7089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:35:39.517225 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:36825
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:39.933991 7104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:39.934335 7104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:39.934479 7104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:39.942222 7104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:39.942509 7104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:39.942620 7104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:39.942719 7104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:39.942788 7104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:39.953655 7104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:39.955292 7104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:39.956682 7104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:39.970423 7111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:39.970471 7110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:39.970508 7113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:39.971565 7104 server_base.cc:1034] running on GCE node
I20260812 02:35:39.972648 7104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:39.974826 7104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:39.976094 7104 hybrid_clock.cc:648] HybridClock initialized: now 1786502139976020 us; error 59 us; skew 500 ppm
I20260812 02:35:39.976559 7104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:39 dist-test-slave-vq2q krb5kdc[7089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502139, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:39.980870 7104 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:35:39.983067 7104 webserver.cc:460] Webserver started at http://127.12.8.254:40439/ using document root <none> and password file <none>
I20260812 02:35:39.984030 7104 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:39.984187 7104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:39.984586 7104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:39.988561 7104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "121d14e3b52e47f7b304eebf61e789a5"
format_stamp: "Formatted at 2026-08-12 02:35:39 on dist-test-slave-vq2q"
server_key: "f236ef58710df9767d1a3812e7e891e5"
server_key_iv: "5a650f102345eff90ae4b0734a028021"
server_key_version: "encryptionkey@0"
I20260812 02:35:39.989675 7104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "121d14e3b52e47f7b304eebf61e789a5"
format_stamp: "Formatted at 2026-08-12 02:35:39 on dist-test-slave-vq2q"
server_key: "f236ef58710df9767d1a3812e7e891e5"
server_key_iv: "5a650f102345eff90ae4b0734a028021"
server_key_version: "encryptionkey@0"
I20260812 02:35:39.996981 7104 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 02:35:40.001919 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:40.003981 7104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:35:40.004248 7104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "121d14e3b52e47f7b304eebf61e789a5"
format_stamp: "Formatted at 2026-08-12 02:35:39 on dist-test-slave-vq2q"
server_key: "f236ef58710df9767d1a3812e7e891e5"
server_key_iv: "5a650f102345eff90ae4b0734a028021"
server_key_version: "encryptionkey@0"
I20260812 02:35:40.004519 7104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:40.012535 7104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:40.018234 7104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:40.018625 7104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:40.043964 7104 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:36825
I20260812 02:35:40.044021 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:36825 every 8 connection(s)
I20260812 02:35:40.045787 7104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:40.050115 7186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:40.054961 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 7104
I20260812 02:35:40.055181 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:40.055781 12323 external_mini_cluster.cc:1428] Setting key d81cc5725b27d35c57301238cdc2bbcf
I20260812 02:35:40.062410 7186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5: Bootstrap starting.
I20260812 02:35:40.066069 7186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:40.066205 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:40.058284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60209 (local address 127.12.8.254:36825)
0812 02:35:40.059397 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:40.059413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:40.059481 (+ 68us) server_negotiation.cc:407] Connection header received
0812 02:35:40.061736 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:40.061769 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:40.062276 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:40.062763 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:40.064086 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:40.064094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:40.064114 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:40.064132 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:40.064222 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:40.064251 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:40.064517 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:40.064668 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 02:35:40.064899 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":493,"threads_started":1}
I20260812 02:35:40.067289 7186 log.cc:824] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:40.070608 7186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5: No bootstrap required, opened a new log
I20260812 02:35:40.071058 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:40.079136 7186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121d14e3b52e47f7b304eebf61e789a5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36825 } }
I20260812 02:35:40.079788 7186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:40.079787 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:40.076515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47483 (local address 127.12.8.254:36825)
0812 02:35:40.076783 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:40.076790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:40.077234 (+ 444us) server_negotiation.cc:407] Connection header received
0812 02:35:40.077524 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:40.077530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:40.077602 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:40.077749 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:40.078967 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:40.078973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:40.078977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:35:40.078997 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:40.079087 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:35:40.079107 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:40.079243 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:40.079382 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 02:35:40.079443 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260812 02:35:40.079923 7186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d14e3b52e47f7b304eebf61e789a5, State: Initialized, Role: FOLLOWER
I20260812 02:35:40.080554 7186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121d14e3b52e47f7b304eebf61e789a5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36825 } }
I20260812 02:35:40.080902 7186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:40.081070 7186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:40.081324 7186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:40.083678 7186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121d14e3b52e47f7b304eebf61e789a5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36825 } }
I20260812 02:35:40.084224 7186 leader_election.cc:302] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 121d14e3b52e47f7b304eebf61e789a5; no voters:
I20260812 02:35:40.084937 7186 leader_election.cc:288] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:40.085351 7196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:40.086315 7196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [term 1 LEADER]: Becoming Leader. State: Replica: 121d14e3b52e47f7b304eebf61e789a5, State: Running, Role: LEADER
I20260812 02:35:40.087213 7196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121d14e3b52e47f7b304eebf61e789a5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36825 } }
I20260812 02:35:40.087736 7186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:40.091256 7197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "121d14e3b52e47f7b304eebf61e789a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121d14e3b52e47f7b304eebf61e789a5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36825 } } }
I20260812 02:35:40.091159 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 121d14e3b52e47f7b304eebf61e789a5. Latest consensus state: current_term: 1 leader_uuid: "121d14e3b52e47f7b304eebf61e789a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121d14e3b52e47f7b304eebf61e789a5" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36825 } } }
I20260812 02:35:40.091657 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:40.091657 7197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:40.092558 7207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:40.098939 7207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:40.112598 7207 catalog_manager.cc:1347] Generated new cluster ID: 177c1bb3b27f4d9e8b40bbd00f6743e4
I20260812 02:35:40.112749 7207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:40.136569 7207 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:40.137246 7207 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:40.145300 7207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 121d14e3b52e47f7b304eebf61e789a5: Generated new TSK 0
I20260812 02:35:40.146508 7207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:35:40.164553 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 7104
W20260812 02:35:40.166178 7179 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60209
W20260812 02:35:40.167229 12323 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 7104: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 02:35:40.167438 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 7104
2026-08-12T02:35:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2293 ms)
[----------] 18 tests from AuthTokenIssuingTest (41809 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:40 dist-test-slave-vq2q krb5kdc[7222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:40 dist-test-slave-vq2q krb5kdc[7222](info): set up 2 sockets
Aug 12 02:35:40 dist-test-slave-vq2q krb5kdc[7222](info): commencing operation
krb5kdc: starting...
W20260812 02:35:41.912709 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.697s user 0.003s sys 0.092s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:42 dist-test-slave-vq2q krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:35:42.357439 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:36501
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:36501,127.12.8.253:39817,127.12.8.252:35599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:42.807857 7243 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:35:42.808265 7243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:42.808378 7243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:42.808508 7243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:42.816226 7243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:42.816368 7243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:42.816459 7243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:42.816538 7243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:42.816628 7243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:42.827414 7243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:36501,127.12.8.253:39817,127.12.8.252:35599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:42.828972 7243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:42.830618 7243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:42.843997 7249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:42.844036 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:42.843899 7248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:42.845355 7243 server_base.cc:1034] running on GCE node
I20260812 02:35:42.846846 7243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:42.849272 7243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:42.850652 7243 hybrid_clock.cc:648] HybridClock initialized: now 1786502142850537 us; error 62 us; skew 500 ppm
I20260812 02:35:42.851296 7243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:42 dist-test-slave-vq2q krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502142, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:42.855834 7243 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:35:42.858206 7243 webserver.cc:460] Webserver started at http://127.12.8.254:46199/ using document root <none> and password file <none>
I20260812 02:35:42.859218 7243 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:42.859356 7243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:42.859792 7243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:42.864519 7243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "50456bad426a4761a0aa8a1804e98ff3"
format_stamp: "Formatted at 2026-08-12 02:35:42 on dist-test-slave-vq2q"
server_key: "98e9dc244ef2fb83d90d4e14401cfb7f"
server_key_iv: "051efd7a8eed601b5d93a16ad2f91294"
server_key_version: "encryptionkey@0"
I20260812 02:35:42.865806 7243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "50456bad426a4761a0aa8a1804e98ff3"
format_stamp: "Formatted at 2026-08-12 02:35:42 on dist-test-slave-vq2q"
server_key: "98e9dc244ef2fb83d90d4e14401cfb7f"
server_key_iv: "051efd7a8eed601b5d93a16ad2f91294"
server_key_version: "encryptionkey@0"
I20260812 02:35:42.873849 7243 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260812 02:35:42.879134 7261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:42.881366 7243 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260812 02:35:42.881625 7243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "50456bad426a4761a0aa8a1804e98ff3"
format_stamp: "Formatted at 2026-08-12 02:35:42 on dist-test-slave-vq2q"
server_key: "98e9dc244ef2fb83d90d4e14401cfb7f"
server_key_iv: "051efd7a8eed601b5d93a16ad2f91294"
server_key_version: "encryptionkey@0"
I20260812 02:35:42.881915 7243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:42.919607 7243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:42.925339 7243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:42.925698 7243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:42.949223 7243 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:36501
I20260812 02:35:42.949244 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:36501 every 8 connection(s)
I20260812 02:35:42.951078 7243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:42.953054 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 7243
I20260812 02:35:42.953359 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:42.954238 12323 external_mini_cluster.cc:1428] Setting key b2c3f60e64d8d1a9f327643e6a36d155
I20260812 02:35:42.956413 7321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:42.965520 7321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:42 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502142, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:42.994483 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:42.970096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:36501 (local address 127.0.0.1:49876)
0812 02:35:42.971661 (+ 1565us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:42.971722 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:42.973944 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:42.975277 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:42.975284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:42.975940 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:42.976348 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:42.976366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:42.978890 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:42.978894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:42.980066 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:42.980080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:42.981521 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:42.981528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:42.981701 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:42.982420 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:42.982445 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:42.984696 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:42.989565 (+ 4869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:42.989582 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:42.989592 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:42.990095 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:42.990715 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:42.990756 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:42.990758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:42.990942 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:42.992169 (+ 1227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:42.992176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:42.992434 (+ 258us) client_negotiation.cc:769] Sending connection context
0812 02:35:42.992680 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 02:35:42.993142 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"thread_start_us":942,"threads_started":1}
I20260812 02:35:42.994571 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:42.970097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.12.8.254:36501)
0812 02:35:42.971293 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:42.971311 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:42.971895 (+ 584us) server_negotiation.cc:407] Connection header received
0812 02:35:42.974472 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:42.974482 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:42.974920 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:42.975237 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:42.976623 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:42.978638 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:42.980244 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:42.981158 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:42.981423 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:42.985063 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:42.985079 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:42.985105 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:42.985231 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:42.989337 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:42.990277 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:42.990289 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:42.990293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:42.990560 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:42.991062 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:42.991066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:42.991069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:42.991976 (+ 907us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:42.992221 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:42.992894 (+ 673us) server_negotiation.cc:299] Negotiation successful
0812 02:35:42.993402 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":836,"thread_start_us":507,"threads_started":1}
I20260812 02:35:42.998412 7321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:35:42.999899 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:42.999390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35756)
0812 02:35:42.999623 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:42.999741 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260812 02:35:43.000211 7265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.12.8.253@KRBTEST.COM; defaulting to no policy
I20260812 02:35:43.002256 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.001787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35758)
0812 02:35:43.002019 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.002110 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 02:35:43.002512 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:39817: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab.
Entry for principal kudu/127.12.8.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab.
I20260812 02:35:43.051954 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } attempt: 1
I20260812 02:35:43.052671 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.253:39817
--webserver_interface=127.12.8.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab
--principal=kudu/127.12.8.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:36501,127.12.8.253:39817,127.12.8.252:35599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:35:43.053573 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.052946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35774)
0812 02:35:43.053268 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.053391 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 02:35:43.054699 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.054298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35778)
0812 02:35:43.054450 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.054542 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 02:35:43.054888 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:39817: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
I20260812 02:35:43.123191 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } attempt: 2
I20260812 02:35:43.124951 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.124211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35788)
0812 02:35:43.124566 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.124731 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 02:35:43.126432 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.125874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35792)
0812 02:35:43.126067 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.126206 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 02:35:43.126718 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:39817: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
I20260812 02:35:43.218039 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } attempt: 3
I20260812 02:35:43.219739 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.219052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35796)
0812 02:35:43.219357 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.219519 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 02:35:43.220976 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.220498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35804)
0812 02:35:43.220655 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.220762 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 02:35:43.221289 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:39817: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
I20260812 02:35:43.364672 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } attempt: 4
I20260812 02:35:43.366622 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.365698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35806)
0812 02:35:43.366209 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.366405 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302}
I20260812 02:35:43.367988 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.367496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35810)
0812 02:35:43.367671 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.367801 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 02:35:43.368294 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:39817: Network error: Client connection negotiation failed: client connection to 127.12.8.253:39817: connect: Connection refused (error 111)
W20260812 02:35:43.504398 7329 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:35:43.504850 7329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:43.504938 7329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:43.505033 7329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:43.513628 7329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:43.513787 7329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:43.513865 7329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:43.513942 7329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:43.514016 7329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:43.525916 7329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:36501,127.12.8.253:39817,127.12.8.252:35599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab
--principal=kudu/127.12.8.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.253:39817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.8.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:43.527796 7329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:43.529582 7329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:43.542948 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:43.543685 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:43.544641 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:43.546066 7329 server_base.cc:1034] running on GCE node
I20260812 02:35:43.547242 7329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:43.549227 7329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:43.550503 7329 hybrid_clock.cc:648] HybridClock initialized: now 1786502143550420 us; error 46 us; skew 500 ppm
I20260812 02:35:43.550938 7329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:43 dist-test-slave-vq2q krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502143, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:43.554895 7329 init.cc:377] Logged in from keytab as kudu/127.12.8.253@KRBTEST.COM (short username kudu)
I20260812 02:35:43.556918 7329 webserver.cc:460] Webserver started at http://127.12.8.253:41669/ using document root <none> and password file <none>
I20260812 02:35:43.557940 7329 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:43.558066 7329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:43.558461 7329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:43.562299 7329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/data/instance:
uuid: "a6d4fa9977394275aff5dae8c706bc06"
format_stamp: "Formatted at 2026-08-12 02:35:43 on dist-test-slave-vq2q"
server_key: "e94c4e59153f226002be6962593b9a44"
server_key_iv: "c94495983a4f6dec081baf208c00cc1c"
server_key_version: "encryptionkey@0"
I20260812 02:35:43.563342 7329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/wal/instance:
uuid: "a6d4fa9977394275aff5dae8c706bc06"
format_stamp: "Formatted at 2026-08-12 02:35:43 on dist-test-slave-vq2q"
server_key: "e94c4e59153f226002be6962593b9a44"
server_key_iv: "c94495983a4f6dec081baf208c00cc1c"
server_key_version: "encryptionkey@0"
I20260812 02:35:43.570720 7329 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 02:35:43.575542 7349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:43.577716 7329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 02:35:43.577972 7329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/wal
uuid: "a6d4fa9977394275aff5dae8c706bc06"
format_stamp: "Formatted at 2026-08-12 02:35:43 on dist-test-slave-vq2q"
server_key: "e94c4e59153f226002be6962593b9a44"
server_key_iv: "c94495983a4f6dec081baf208c00cc1c"
server_key_version: "encryptionkey@0"
I20260812 02:35:43.578270 7329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:43.602300 7329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:43.607797 7329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:43.608158 7329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:43.630733 7329 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.253:39817
I20260812 02:35:43.630788 7414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.253:39817 every 8 connection(s)
I20260812 02:35:43.632539 7329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
I20260812 02:35:43.637070 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:43.639292 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 7329
I20260812 02:35:43.639521 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-1/wal/instance
I20260812 02:35:43.640106 12323 external_mini_cluster.cc:1428] Setting key c36664733f15084a289443487311b06e
I20260812 02:35:43.646320 7415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:43 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502143, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:43.667588 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } attempt: 5
I20260812 02:35:43.674048 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.650034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.12.8.254:36501)
0812 02:35:43.650719 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:43.650727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:43.652278 (+ 1551us) server_negotiation.cc:407] Connection header received
0812 02:35:43.654995 (+ 2717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:43.655004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:43.655120 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:43.655294 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:43.657874 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.658767 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.660533 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.661162 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.661365 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:43.665415 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:43.665424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:43.665429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:43.665506 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:43.669933 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:43.670980 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:43.670987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:43.670990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:43.671173 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:43.671711 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:43.671717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:43.671720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:43.672155 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:43.672327 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:43.673317 (+ 990us) server_negotiation.cc:299] Negotiation successful
0812 02:35:43.673652 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":309,"threads_started":1}
I20260812 02:35:43.675419 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.650924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:36501 (local address 127.0.0.1:49878)
0812 02:35:43.652039 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.652104 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:43.654544 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:43.655627 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:43.655639 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:43.656474 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:43.657400 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:43.657428 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.659006 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.659011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:43.660343 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:43.660359 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.661460 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.661466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:43.661685 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:43.662389 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:43.662420 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:43.665022 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:43.670173 (+ 5151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:43.670182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:43.670197 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:43.670836 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:43.671308 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:43.671313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:43.671316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:43.671562 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:43.672312 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:43.672319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:43.672768 (+ 449us) client_negotiation.cc:769] Sending connection context
0812 02:35:43.673097 (+ 329us) client_negotiation.cc:240] Negotiation successful
0812 02:35:43.673665 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":424,"threads_started":1}
Aug 12 02:35:43 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502142, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:35:43.678004 7415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.8.252@KRBTEST.COM; defaulting to no policy
I20260812 02:35:43.687135 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.668315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35822)
0812 02:35:43.668580 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.668604 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:43.668771 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:43.669897 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:43.669903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:43.670465 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:43.670709 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:43.670722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.672405 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.672410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:43.673794 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:43.673809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.674960 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.674966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:43.675192 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:43.675931 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:43.675954 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:43.678670 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:43.683634 (+ 4964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:43.683640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:43.683644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:43.684150 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:43.684818 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:43.684823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:43.684827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:43.685006 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:43.686046 (+ 1040us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:43.686052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:43.686262 (+ 210us) client_negotiation.cc:769] Sending connection context
0812 02:35:43.686429 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 02:35:43.686717 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 02:35:43.688270 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.668637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.12.8.253:39817)
0812 02:35:43.669412 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:43.669419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:43.669451 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:35:43.669500 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:43.669506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:43.669705 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:43.669852 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:43.670955 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.672232 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.673984 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.674784 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.675543 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:43.679052 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:43.679077 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:43.679098 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:43.679151 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:43.683469 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:43.684366 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:43.684379 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:43.684383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:43.684668 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:43.685157 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:43.685163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:43.685166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:43.685885 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:43.686180 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:43.687558 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0812 02:35:43.687859 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":407,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.252@KRBTEST.COM" created.
Aug 12 02:35:43 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502143, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:35:43.691597 7321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:35:43.693267 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.692317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50794)
0812 02:35:43.692595 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.692735 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 02:35:43.694600 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.694132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50802)
0812 02:35:43.694379 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.694463 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260812 02:35:43.694885 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:43.697093 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.678674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35830)
0812 02:35:43.678961 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.678981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:43.679100 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:43.681091 (+ 1991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:43.681096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:43.682014 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:43.682260 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:43.682280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.684252 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.684257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:43.685457 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:43.685472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.686368 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.686375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:43.686589 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:43.687344 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:43.687368 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:43.691000 (+ 3632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:43.694616 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:43.694622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:43.694627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:43.695115 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:43.695487 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:43.695492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:43.695496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:43.695621 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:43.696156 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:43.696161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:43.696326 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 02:35:43.696434 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 02:35:43.696729 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 02:35:43.697358 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.678963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.12.8.253:39817)
0812 02:35:43.680669 (+ 1706us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:43.680674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:43.680706 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:35:43.680772 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:43.680777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:43.680865 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:43.681004 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:43.682508 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.684075 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.685595 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:43.686161 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:43.686308 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:43.691239 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:43.691244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:43.691248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:43.691285 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:43.694479 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:43.695237 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:43.695241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:43.695243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:43.695372 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:43.695739 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:43.695742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:43.695744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:43.696043 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:43.696181 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:43.696554 (+ 373us) server_negotiation.cc:299] Negotiation successful
0812 02:35:43.696871 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":1506,"thread_start_us":1190,"threads_started":1}
I20260812 02:35:43.698617 7415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:35:43.699927 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.699333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50816)
0812 02:35:43.699564 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.699715 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260812 02:35:43.700240 7351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:43.702245 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.701875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50830)
0812 02:35:43.702052 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.702140 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 02:35:43.702483 7415 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab.
Entry for principal kudu/127.12.8.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab.
I20260812 02:35:43.739287 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.252:35599
--webserver_interface=127.12.8.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab
--principal=kudu/127.12.8.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:36501,127.12.8.253:39817,127.12.8.252:35599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:35:43.746146 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 1
I20260812 02:35:43.747788 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.747100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50838)
0812 02:35:43.747414 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.747545 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260812 02:35:43.749119 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.748722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50846)
0812 02:35:43.748940 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.749008 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260812 02:35:43.749396 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:43.751889 7415 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 1
I20260812 02:35:43.753574 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.752796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50856)
0812 02:35:43.753086 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.753291 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 02:35:43.755110 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.754608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50868)
0812 02:35:43.754799 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.754939 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 02:35:43.755425 7415 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:43.817643 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 2
I20260812 02:35:43.819422 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.818674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50876)
0812 02:35:43.818993 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.819178 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260812 02:35:43.820643 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.820249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50878)
0812 02:35:43.820417 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.820505 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 02:35:43.820868 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:43.823742 7415 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 2
I20260812 02:35:43.825296 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.824623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50886)
0812 02:35:43.824890 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.825013 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 02:35:43.826565 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.826161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50902)
0812 02:35:43.826318 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.826416 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 02:35:43.826854 7415 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:43.918183 7415 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 3
I20260812 02:35:43.919957 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.919157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50918)
0812 02:35:43.919590 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.919738 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260812 02:35:43.921363 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.920779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50932)
0812 02:35:43.921043 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.921140 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260812 02:35:43.921598 7415 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:43.927196 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 3
I20260812 02:35:43.928799 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.928161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50946)
0812 02:35:43.928422 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.928540 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 02:35:43.930143 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:43.929696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50948)
0812 02:35:43.929892 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:43.930012 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 02:35:43.930414 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:44.064921 7415 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 4
I20260812 02:35:44.066537 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.065941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50952)
0812 02:35:44.066220 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.066325 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 02:35:44.067710 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.067268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50966)
0812 02:35:44.067454 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.067552 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 02:35:44.067957 7415 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
I20260812 02:35:44.107736 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 4
I20260812 02:35:44.109433 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.108775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50972)
0812 02:35:44.109028 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.109141 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 02:35:44.110733 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.110294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:50982)
0812 02:35:44.110478 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.110572 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 02:35:44.110996 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:35599: Network error: Client connection negotiation failed: client connection to 127.12.8.252:35599: connect: Connection refused (error 111)
W20260812 02:35:44.144395 7430 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:35:44.144760 7430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:44.144845 7430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:44.144985 7430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:44.152570 7430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:44.152674 7430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:44.152732 7430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:44.152788 7430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:44.152892 7430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:44.163332 7430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:36501,127.12.8.253:39817,127.12.8.252:35599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab
--principal=kudu/127.12.8.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.252:35599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.8.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:44.164834 7430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:44.166339 7430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:44.179381 7438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:44.179490 7436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:44.179576 7435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:44.181159 7430 server_base.cc:1034] running on GCE node
I20260812 02:35:44.182096 7430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:44.183772 7430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:44.185029 7430 hybrid_clock.cc:648] HybridClock initialized: now 1786502144184944 us; error 63 us; skew 500 ppm
I20260812 02:35:44.185424 7430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502144, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:44.189635 7430 init.cc:377] Logged in from keytab as kudu/127.12.8.252@KRBTEST.COM (short username kudu)
I20260812 02:35:44.191677 7430 webserver.cc:460] Webserver started at http://127.12.8.252:33127/ using document root <none> and password file <none>
I20260812 02:35:44.192852 7430 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:44.193010 7430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:44.193487 7430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:44.197983 7430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/data/instance:
uuid: "49e28f25237244c4a1a84d389f0478c8"
format_stamp: "Formatted at 2026-08-12 02:35:44 on dist-test-slave-vq2q"
server_key: "21b9826f6187e537191f9517be126c1e"
server_key_iv: "4d5cfadcb9d3506bfec4836af269ba8b"
server_key_version: "encryptionkey@0"
I20260812 02:35:44.199213 7430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/wal/instance:
uuid: "49e28f25237244c4a1a84d389f0478c8"
format_stamp: "Formatted at 2026-08-12 02:35:44 on dist-test-slave-vq2q"
server_key: "21b9826f6187e537191f9517be126c1e"
server_key_iv: "4d5cfadcb9d3506bfec4836af269ba8b"
server_key_version: "encryptionkey@0"
I20260812 02:35:44.206645 7430 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 02:35:44.211582 7447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:44.213634 7430 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:35:44.213884 7430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/wal
uuid: "49e28f25237244c4a1a84d389f0478c8"
format_stamp: "Formatted at 2026-08-12 02:35:44 on dist-test-slave-vq2q"
server_key: "21b9826f6187e537191f9517be126c1e"
server_key_iv: "4d5cfadcb9d3506bfec4836af269ba8b"
server_key_version: "encryptionkey@0"
I20260812 02:35:44.214185 7430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:44.229507 7430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:44.235177 7430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:44.235488 7430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:44.257512 7430 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.252:35599
I20260812 02:35:44.257537 7505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.252:35599 every 8 connection(s)
I20260812 02:35:44.259255 7430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
I20260812 02:35:44.260474 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 7430
I20260812 02:35:44.260681 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501741404412-12323-0/minicluster-data/master-2/wal/instance
I20260812 02:35:44.261155 12323 external_mini_cluster.cc:1428] Setting key 0b93a8454badcf1d3335bf3d94384634
I20260812 02:35:44.264225 7506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:44.273502 7506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502144, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:44.290292 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.263653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.12.8.252:35599)
0812 02:35:44.264651 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.264670 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.264733 (+ 63us) server_negotiation.cc:407] Connection header received
0812 02:35:44.267159 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.267189 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.267758 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.268250 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.270276 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.272289 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.274035 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.275084 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.275342 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.280971 (+ 5629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.280982 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.280996 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.281045 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.285302 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.286403 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.286410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.286416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.286580 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.287061 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.287066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.287070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.287741 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.288051 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.288627 (+ 576us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.289229 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":391,"threads_started":1}
I20260812 02:35:44.294343 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.276867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:36501 (local address 127.0.0.1:49890)
0812 02:35:44.277856 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.277904 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:44.278087 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:44.278472 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:44.278480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:44.279297 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:44.279679 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.279694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.280740 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.280745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.282373 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.282391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.283585 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.283593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.283776 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.284531 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:44.284557 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:44.287443 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:44.291199 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.291229 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.291238 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.291817 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.292265 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.292271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.292279 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.292456 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.293239 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:44.293247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:44.293467 (+ 220us) client_negotiation.cc:769] Sending connection context
0812 02:35:44.293653 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 02:35:44.293958 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":576,"threads_started":1}
I20260812 02:35:44.295394 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.276769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.12.8.254:36501)
0812 02:35:44.277395 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.277402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.278003 (+ 601us) server_negotiation.cc:407] Connection header received
0812 02:35:44.278214 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.278220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.278285 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.278389 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.279838 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.280588 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.282675 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.283361 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.283504 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.287666 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.287672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.287676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.287742 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.290990 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.291974 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.291979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.291982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.292133 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.292579 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.292583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.292586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.293046 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.293285 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.294718 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.295012 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":299,"threads_started":1}
I20260812 02:35:44.297097 7506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502144, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:35:44.311250 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.293606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.12.8.254:36501)
0812 02:35:44.294388 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.294394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.294421 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:35:44.294478 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.294483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.294563 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.294675 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.296474 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.297388 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.298955 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.299644 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.299816 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.304572 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.304578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.304582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.304628 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.307933 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.308789 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.308793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.308796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.308970 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.309438 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.309441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.309444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.309802 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.310099 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.310544 (+ 445us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.310823 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":357,"threads_started":1}
I20260812 02:35:44.312319 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.298106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:39817 (local address 127.0.0.1:35834)
0812 02:35:44.298354 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.298379 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:44.298519 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:44.299187 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:44.299192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:44.299784 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:44.300014 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.300025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.301368 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.301373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.302389 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.302402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.303249 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.303257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.303480 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.304051 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:44.304075 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:44.306433 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:44.309819 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.309825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.309830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.310217 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.310583 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.310589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.310593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.310738 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.311276 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:44.311281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:44.311491 (+ 210us) client_negotiation.cc:769] Sending connection context
0812 02:35:44.311655 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 02:35:44.311966 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 02:35:44.312384 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.298097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.12.8.253:39817)
0812 02:35:44.298847 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.298853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.298875 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:35:44.298944 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.298949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.299021 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.299110 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.300187 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.301222 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.302558 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.303054 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.303169 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.306643 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.306648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.306652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.306685 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.309683 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.310337 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.310343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.310345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.310473 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.310848 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.310853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.310855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.311173 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.311312 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.311839 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.312072 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":426,"threads_started":1}
I20260812 02:35:44.313604 7506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502144, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:35:44.328029 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.314451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.12.8.252:35599)
0812 02:35:44.314640 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.314647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.314662 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:35:44.314697 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.314700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.314763 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.314837 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.315536 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.316244 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.317412 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.317845 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.317948 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.321723 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.321729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.321732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.321760 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.324661 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.325659 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.325664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.325666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.325893 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.326359 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.326362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.326364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.326650 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.326765 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.327305 (+ 540us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.327591 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":71}
I20260812 02:35:44.328253 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.314280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:51008)
0812 02:35:44.314439 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.314458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:44.314558 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:44.314877 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:44.314881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:44.315273 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:44.315409 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.315419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.316368 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.316373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.317290 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.317303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.317992 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.317997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.318166 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.318874 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:44.318901 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:44.321514 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:44.324846 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.324852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.324856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.325509 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.326041 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.326047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.326051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.326234 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.326778 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:44.326784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:44.327005 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 02:35:44.327184 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 02:35:44.327772 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":176}
I20260812 02:35:44.330121 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.313233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.12.8.253:39817)
0812 02:35:44.313404 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.313408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.313558 (+ 150us) server_negotiation.cc:407] Connection header received
0812 02:35:44.313755 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.313760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.313835 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.313928 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.315439 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.316211 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.317561 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.318059 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.318177 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.323281 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.323287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.323290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.323332 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.326530 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.327557 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.327564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.327567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.327769 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.328293 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.328299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.328303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.328762 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.329090 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.329500 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.329752 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:35:44.335098 7506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: Bootstrap starting.
I20260812 02:35:44.335566 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:44.339351 7506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:44.340752 7506 log.cc:824] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:44.338490 7293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "50456bad426a4761a0aa8a1804e98ff3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49896
I20260812 02:35:44.344631 7506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: No bootstrap required, opened a new log
I20260812 02:35:44.353374 7506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:35:44.353839 7506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e28f25237244c4a1a84d389f0478c8, State: Initialized, Role: FOLLOWER
I20260812 02:35:44.354413 7506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:35:44.355746 7517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.356274 7517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:44.356925 7506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:35:44.367745 7529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:35:44.367918 7529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:35:44.369326 7415 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 5
I20260812 02:35:44.366339 7430 master_runner.cc:186] Error getting master registration for 127.12.8.254:36501: OK, instance_id { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" instance_seqno: 1786502142942714 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:35:44.378724 7430 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502143, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
I20260812 02:35:44.386207 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.370073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:51024)
0812 02:35:44.370362 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.370383 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:44.370502 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:44.371285 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:44.371291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:44.371852 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:44.372081 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.372094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.373823 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.373830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.375259 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.375274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.376097 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.376102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.376310 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.376935 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:44.376961 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:44.380070 (+ 3109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:44.383634 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.383639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.383643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.384056 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.384433 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.384438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.384442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.384579 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.385205 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:44.385211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:44.385362 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 02:35:44.385483 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 02:35:44.385821 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260812 02:35:44.386209 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.370150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.12.8.252:35599)
0812 02:35:44.370758 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.370811 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.370830 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:35:44.370897 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.370903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.370982 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.371076 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.372240 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.373384 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.375408 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.375942 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.376055 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.380285 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.380295 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.380298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.380333 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.383469 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.384180 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.384187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.384190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.384326 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.384686 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.384696 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.384700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.385058 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.385241 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.385608 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.385832 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260812 02:35:44.388271 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } attempt: 5
I20260812 02:35:44.392027 7415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06: Bootstrap starting.
I20260812 02:35:44.395824 7415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:44.397197 7415 log.cc:824] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:35:44 dist-test-slave-vq2q krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502142, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
I20260812 02:35:44.401782 7415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06: No bootstrap required, opened a new log
I20260812 02:35:44.413300 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.389084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:35599 (local address 127.0.0.1:51028)
0812 02:35:44.389381 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:44.389407 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:44.389591 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:44.390010 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:44.390016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:44.390575 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:44.390820 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.390834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.392506 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.392513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.394236 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:44.394260 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.395338 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.395344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:44.395563 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.396453 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:44.396480 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:44.406143 (+ 9663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:44.410244 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.410250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.410254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.410832 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.411241 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:44.411246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:44.411250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:44.411454 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:44.412129 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:44.412133 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:44.412371 (+ 238us) client_negotiation.cc:769] Sending connection context
0812 02:35:44.412512 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 02:35:44.412838 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 02:35:44.413300 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.389421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.12.8.252:35599)
0812 02:35:44.389602 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.389607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.389633 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:35:44.389710 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.389715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.389823 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.390978 (+ 1155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.391122 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.392267 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.394480 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.395138 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.395291 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.406491 (+ 11200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.406497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.406500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.406539 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.410053 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.410967 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.410973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.410977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.411131 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.411608 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.411614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.411633 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.411997 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.412149 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.412670 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.412899 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:35:44.414343 7415 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:35:44.414937 7415 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d4fa9977394275aff5dae8c706bc06, State: Initialized, Role: FOLLOWER
I20260812 02:35:44.415683 7415 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:35:44.417369 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.418325 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:44.418989 7415 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:44.420207 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3: Bootstrap starting.
I20260812 02:35:44.424547 7321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:44.426051 7321 log.cc:824] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:44.429896 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3: No bootstrap required, opened a new log
I20260812 02:35:44.437618 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:35:44.438056 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50456bad426a4761a0aa8a1804e98ff3, State: Initialized, Role: FOLLOWER
I20260812 02:35:44.438572 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:35:44.439766 7542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.440510 7542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:44.441210 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:44.441320 7293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 02:35:44.441506 7293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:44.445140 7293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
I20260812 02:35:44.448246 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50456bad426a4761a0aa8a1804e98ff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a6d4fa9977394275aff5dae8c706bc06"
I20260812 02:35:44.448836 7381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:44.453089 7381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50456bad426a4761a0aa8a1804e98ff3 in term 1.
I20260812 02:35:44.454074 7265 leader_election.cc:302] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50456bad426a4761a0aa8a1804e98ff3, a6d4fa9977394275aff5dae8c706bc06; no voters:
I20260812 02:35:44.455981 7478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50456bad426a4761a0aa8a1804e98ff3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "49e28f25237244c4a1a84d389f0478c8"
I20260812 02:35:44.456640 7478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:44.461306 7478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50456bad426a4761a0aa8a1804e98ff3 in term 1.
I20260812 02:35:44.462190 7293 leader_election.cc:288] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [CANDIDATE]: Term 1 election: Requested vote from peers a6d4fa9977394275aff5dae8c706bc06 (127.12.8.253:39817), 49e28f25237244c4a1a84d389f0478c8 (127.12.8.252:35599)
I20260812 02:35:44.469547 7542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:44.479424 7542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [term 1 LEADER]: Becoming Leader. State: Replica: 50456bad426a4761a0aa8a1804e98ff3, State: Running, Role: LEADER
I20260812 02:35:44.479890 7556 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:35:44.481431 7542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } }
W20260812 02:35:44.488245 7453 tablet.cc:2369] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:35:44.483342 7329 master_runner.cc:186] Error getting master registration for 127.12.8.254:36501: OK, instance_id { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" instance_seqno: 1786502142942714 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:35:44.495357 7329 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:35:44.496546 7555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50456bad426a4761a0aa8a1804e98ff3. Latest consensus state: current_term: 1 leader_uuid: "50456bad426a4761a0aa8a1804e98ff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.497143 7555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:44.498525 7568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:44.512589 7568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:44.534904 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.494095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.12.8.253:39817)
0812 02:35:44.494401 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.494410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.499466 (+ 5056us) server_negotiation.cc:407] Connection header received
0812 02:35:44.509465 (+ 9999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.509475 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.509610 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.509729 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.512888 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.514051 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.518857 (+ 4806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.519613 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.519756 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.523441 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.523448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.523452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.523502 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.527172 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.531808 (+ 4636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.531814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.531818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.532022 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.532494 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.532499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.532502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.532942 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.533255 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.534247 (+ 992us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.534514 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:35:44.540485 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.502840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.12.8.252:35599)
0812 02:35:44.503146 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.503153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.511955 (+ 8802us) server_negotiation.cc:407] Connection header received
0812 02:35:44.513530 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.513537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.513633 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.513705 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.520279 (+ 6574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.521340 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.523391 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.524013 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.524110 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.529139 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.529146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.529150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.529215 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.534941 (+ 5726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.537815 (+ 2874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.537821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.537825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.537990 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.538515 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.538521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.538525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.538937 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.539199 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.539776 (+ 577us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.540057 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:35:44.547487 7478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [term 1 FOLLOWER]: Refusing update from remote peer 50456bad426a4761a0aa8a1804e98ff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:35:44.548679 7542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:35:44.550524 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:44.498456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.12.8.254:36501)
0812 02:35:44.498713 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.498725 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.510991 (+ 12266us) server_negotiation.cc:407] Connection header received
0812 02:35:44.512371 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.512378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.512487 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.512575 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.536249 (+ 23674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.537357 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.539205 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:44.539919 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:44.540017 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:44.543591 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:44.543596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:44.543600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:44.543636 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:44.547103 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.547887 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.547892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.547895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.548046 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:44.548509 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:44.548514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:44.548516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:44.548957 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:44.549253 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:44.549823 (+ 570us) server_negotiation.cc:299] Negotiation successful
0812 02:35:44.550171 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:35:44.556965 7381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [term 1 FOLLOWER]: Refusing update from remote peer 50456bad426a4761a0aa8a1804e98ff3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:35:44.557847 7542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:35:44.569900 7517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50456bad426a4761a0aa8a1804e98ff3. Latest consensus state: current_term: 1 leader_uuid: "50456bad426a4761a0aa8a1804e98ff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.570320 7517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:44.576336 7575 mvcc.cc:205] Tried to move back new op lower bound from 7317512784028180480 to 7317512783815102464. Current Snapshot: MvccSnapshot[applied={T|T < 7317512784028180480}]
I20260812 02:35:44.576716 7574 mvcc.cc:205] Tried to move back new op lower bound from 7317512784028180480 to 7317512783815102464. Current Snapshot: MvccSnapshot[applied={T|T < 7317512784028180480}]
I20260812 02:35:44.587891 7542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50456bad426a4761a0aa8a1804e98ff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.588394 7542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:44.590768 7568 catalog_manager.cc:1347] Generated new cluster ID: e17c42558b89475387faced50adfffc4
I20260812 02:35:44.590909 7568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:44.592518 7517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50456bad426a4761a0aa8a1804e98ff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.592799 7517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:44.595510 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50456bad426a4761a0aa8a1804e98ff3. Latest consensus state: current_term: 1 leader_uuid: "50456bad426a4761a0aa8a1804e98ff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.595853 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:44.597977 7542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50456bad426a4761a0aa8a1804e98ff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.598390 7542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:44.604725 7577 mvcc.cc:205] Tried to move back new op lower bound from 7317512784028180480 to 7317512783815102464. Current Snapshot: MvccSnapshot[applied={T|T < 7317512784028180480}]
I20260812 02:35:44.622233 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50456bad426a4761a0aa8a1804e98ff3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50456bad426a4761a0aa8a1804e98ff3" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 36501 } } peers { permanent_uuid: "a6d4fa9977394275aff5dae8c706bc06" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 39817 } } peers { permanent_uuid: "49e28f25237244c4a1a84d389f0478c8" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 35599 } } }
I20260812 02:35:44.622581 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:44.627806 7568 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:44.634007 7568 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:44.655040 7568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50456bad426a4761a0aa8a1804e98ff3: Generated new TSK 0
I20260812 02:35:44.661079 7568 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:35:44.799352 7514 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.786605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.12.8.254:36501)
0812 02:35:44.786825 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.786832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.796855 (+ 10023us) server_negotiation.cc:407] Connection header received
0812 02:35:44.797261 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.797269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.797376 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.797468 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.799057 (+ 1589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49912: BlockingRecv error: recv got EOF from 127.0.0.1:49912 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 02:35:44.801978 7511 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.797265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.12.8.252:35599)
0812 02:35:44.797562 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.797569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.800201 (+ 2632us) server_negotiation.cc:407] Connection header received
0812 02:35:44.800339 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.800352 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.800527 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.800728 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.801682 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51042: BlockingRecv error: recv got EOF from 127.0.0.1:51042 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260812 02:35:44.803378 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.787015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.12.8.253:39817)
0812 02:35:44.787217 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.787223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.799301 (+ 12078us) server_negotiation.cc:407] Connection header received
0812 02:35:44.799408 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.799414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.799505 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.799603 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.803138 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35868: BlockingRecv error: recv got EOF from 127.0.0.1:35868 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 02:35:44.814225 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.811778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.12.8.253:39817)
0812 02:35:44.811999 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.812006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.812537 (+ 531us) server_negotiation.cc:407] Connection header received
0812 02:35:44.812773 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.812781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.812917 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.813036 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.813971 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35874: BlockingRecv error: recv got EOF from 127.0.0.1:35874 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 02:35:44.822289 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.820096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.12.8.253:39817)
0812 02:35:44.820261 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.820267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.820747 (+ 480us) server_negotiation.cc:407] Connection header received
0812 02:35:44.820944 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.820950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.821055 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.821168 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.822055 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35882: BlockingRecv error: recv got EOF from 127.0.0.1:35882 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 02:35:44.833520 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.831144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.12.8.253:39817)
0812 02:35:44.831324 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.831330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.831830 (+ 500us) server_negotiation.cc:407] Connection header received
0812 02:35:44.832037 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.832044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.832138 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.832255 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.833215 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35890: BlockingRecv error: recv got EOF from 127.0.0.1:35890 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 02:35:44.845799 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.843325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.12.8.253:39817)
0812 02:35:44.843568 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.843574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.844065 (+ 491us) server_negotiation.cc:407] Connection header received
0812 02:35:44.844281 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.844288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.844474 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.844612 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.845511 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35906: BlockingRecv error: recv got EOF from 127.0.0.1:35906 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 02:35:44.865031 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.862360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.12.8.253:39817)
0812 02:35:44.862588 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.862595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.863131 (+ 536us) server_negotiation.cc:407] Connection header received
0812 02:35:44.863466 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.863472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.863561 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.863642 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.864774 (+ 1132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35916: BlockingRecv error: recv got EOF from 127.0.0.1:35916 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 02:35:44.889843 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.887001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.12.8.253:39817)
0812 02:35:44.887445 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.887452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.888074 (+ 622us) server_negotiation.cc:407] Connection header received
0812 02:35:44.888438 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.888444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.888538 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.888657 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.889599 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35922: BlockingRecv error: recv got EOF from 127.0.0.1:35922 (error 108)
Metrics: {"server-negotiator.queue_time_us":308}
W20260812 02:35:44.928460 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:44.926285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.12.8.253:39817)
0812 02:35:44.926521 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:44.926528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:44.926926 (+ 398us) server_negotiation.cc:407] Connection header received
0812 02:35:44.927096 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:44.927102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:44.927202 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:44.927344 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:44.928174 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35924: BlockingRecv error: recv got EOF from 127.0.0.1:35924 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 02:35:45.008344 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:45.006457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.12.8.253:39817)
0812 02:35:45.006687 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:45.006693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:45.007005 (+ 312us) server_negotiation.cc:407] Connection header received
0812 02:35:45.007201 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:45.007206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:45.007282 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:45.007374 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:45.008170 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35934: BlockingRecv error: recv got EOF from 127.0.0.1:35934 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260812 02:35:45.142614 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:45.140615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.12.8.253:39817)
0812 02:35:45.140863 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:45.140870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:45.141130 (+ 260us) server_negotiation.cc:407] Connection header received
0812 02:35:45.141383 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:45.141388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:45.141459 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:45.141546 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:45.142351 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35940: BlockingRecv error: recv got EOF from 127.0.0.1:35940 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 02:35:45.370335 7529 catalog_manager.cc:1259] Loaded cluster ID: e17c42558b89475387faced50adfffc4
I20260812 02:35:45.370453 7529 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: loading cluster ID for follower catalog manager: success
I20260812 02:35:45.372948 7529 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: acquiring CA information for follower catalog manager: success
I20260812 02:35:45.374790 7529 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49e28f25237244c4a1a84d389f0478c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 02:35:45.404289 7515 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:45.402629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.12.8.253:39817)
0812 02:35:45.402799 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:45.402804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:45.403029 (+ 225us) server_negotiation.cc:407] Connection header received
0812 02:35:45.403194 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:45.403199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:45.403277 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:45.403374 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:45.404111 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35956: BlockingRecv error: recv got EOF from 127.0.0.1:35956 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:35:45.485152 7556 catalog_manager.cc:1259] Loaded cluster ID: e17c42558b89475387faced50adfffc4
I20260812 02:35:45.485316 7556 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06: loading cluster ID for follower catalog manager: success
I20260812 02:35:45.488178 7556 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06: acquiring CA information for follower catalog manager: success
I20260812 02:35:45.490165 7556 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6d4fa9977394275aff5dae8c706bc06: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:35:45.929648 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:45.921757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.12.8.253:39817)
0812 02:35:45.922408 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:45.922413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:45.922438 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:35:45.922659 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:45.922664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:45.922885 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:45.923023 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:45.923614 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.924548 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.926496 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.927002 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.927136 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:45.927836 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:45.928777 (+ 941us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:45.928930 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:45.929160 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 02:35:45.929332 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":354,"threads_started":1}
I20260812 02:35:45.945371 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:45.937916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.12.8.252:35599)
0812 02:35:45.938465 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:45.938471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:45.938523 (+ 52us) server_negotiation.cc:407] Connection header received
0812 02:35:45.938786 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:45.938791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:45.938997 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:45.939189 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:45.939719 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.940501 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.942317 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.942837 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.942962 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:45.943741 (+ 779us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:45.944480 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:45.944629 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:45.944817 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 02:35:45.944923 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":294,"threads_started":1}
I20260812 02:35:45.963455 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:45.954991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.12.8.253:39817)
0812 02:35:45.955306 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:45.955315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:45.956174 (+ 859us) server_negotiation.cc:407] Connection header received
0812 02:35:45.956416 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:45.956422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:45.956504 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:45.956592 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:45.957132 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.958354 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.960837 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.961636 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.961789 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:45.962415 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:45.962618 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:45.962733 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:45.962901 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 02:35:45.963026 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 02:35:45.966152 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:45.958143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.12.8.252:35599)
0812 02:35:45.958712 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:45.958719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:45.959160 (+ 441us) server_negotiation.cc:407] Connection header received
0812 02:35:45.959276 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:45.959282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:45.959394 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:45.959513 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:45.960399 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.961612 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.963821 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.964567 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.964713 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:45.965286 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:45.965439 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:45.965528 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:45.965683 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 02:35:45.965807 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260812 02:35:45.966562 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:45.954831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.12.8.254:36501)
0812 02:35:45.955697 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:45.955704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:45.957952 (+ 2248us) server_negotiation.cc:407] Connection header received
0812 02:35:45.958123 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:45.958129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:45.958409 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:45.959427 (+ 1018us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:45.960183 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.961063 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.963830 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:45.964347 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:45.964522 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:45.965161 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:45.965825 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:45.965943 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:45.966119 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 02:35:45.966237 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":587,"threads_started":1}
I20260812 02:35:45.978967 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 7243
I20260812 02:35:45.980917 7316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:46.108074 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 7243
W20260812 02:35:46.138547 7351 connection.cc:535] server connection from 127.0.0.1:35822 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 02:35:46.139220 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 7329
I20260812 02:35:46.140540 7408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:46.268247 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 7329
I20260812 02:35:46.297324 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 7430
I20260812 02:35:46.298813 7501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:46.427139 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 7430
2026-08-12T02:35:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (6269 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:46 dist-test-slave-vq2q krb5kdc[7688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:46 dist-test-slave-vq2q krb5kdc[7688](info): set up 2 sockets
Aug 12 02:35:46 dist-test-slave-vq2q krb5kdc[7688](info): commencing operation
krb5kdc: starting...
W20260812 02:35:47.666910 12323 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.001s sys 0.084s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/test-user.keytab.
Aug 12 02:35:47 dist-test-slave-vq2q krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
Entry for principal kudu/127.12.8.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab.
I20260812 02:35:48.121294 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40809
--webserver_interface=127.12.8.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40809,127.12.8.253:34897,127.12.8.252:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:48.578649 7703 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:35:48.579162 7703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:48.579282 7703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:48.579412 7703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:48.588521 7703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:48.588687 7703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:48.588784 7703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:48.588861 7703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:48.588961 7703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:48.601339 7703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40809,127.12.8.253:34897,127.12.8.252:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.254.keytab
--principal=kudu/127.12.8.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.254:40809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:48.603238 7703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:48.605008 7703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:48.620375 7711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:48.620649 7709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:48.620644 7713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:48.622565 7703 server_base.cc:1034] running on GCE node
I20260812 02:35:48.623728 7703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:48.625875 7703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:48.627259 7703 hybrid_clock.cc:648] HybridClock initialized: now 1786502148627162 us; error 62 us; skew 500 ppm
I20260812 02:35:48.627761 7703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:48 dist-test-slave-vq2q krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502148, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:48.632118 7703 init.cc:377] Logged in from keytab as kudu/127.12.8.254@KRBTEST.COM (short username kudu)
I20260812 02:35:48.634178 7703 webserver.cc:460] Webserver started at http://127.12.8.254:38363/ using document root <none> and password file <none>
I20260812 02:35:48.635026 7703 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:48.635147 7703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:48.635526 7703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:48.638917 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/data/instance:
uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4"
format_stamp: "Formatted at 2026-08-12 02:35:48 on dist-test-slave-vq2q"
server_key: "164bb0d7d5992586328df3bf9c63f0aa"
server_key_iv: "0cc170e6d539d6166e8238c2c1cbaa2a"
server_key_version: "encryptionkey@0"
I20260812 02:35:48.639719 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/wal/instance:
uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4"
format_stamp: "Formatted at 2026-08-12 02:35:48 on dist-test-slave-vq2q"
server_key: "164bb0d7d5992586328df3bf9c63f0aa"
server_key_iv: "0cc170e6d539d6166e8238c2c1cbaa2a"
server_key_version: "encryptionkey@0"
I20260812 02:35:48.646129 7703 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 02:35:48.650853 7721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:48.652880 7703 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:35:48.653110 7703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/wal
uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4"
format_stamp: "Formatted at 2026-08-12 02:35:48 on dist-test-slave-vq2q"
server_key: "164bb0d7d5992586328df3bf9c63f0aa"
server_key_iv: "0cc170e6d539d6166e8238c2c1cbaa2a"
server_key_version: "encryptionkey@0"
I20260812 02:35:48.653402 7703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:48.668401 7703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:48.673599 7703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:48.673889 7703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:48.696792 7703 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.254:40809
I20260812 02:35:48.696887 7787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.254:40809 every 8 connection(s)
I20260812 02:35:48.698455 7703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:48.704101 7788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:48.707346 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 7703
I20260812 02:35:48.707615 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-0/wal/instance
I20260812 02:35:48.708293 12323 external_mini_cluster.cc:1428] Setting key 3c619afdffb30fac18a7d995b649da80
I20260812 02:35:48.715703 7788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:48 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502148, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:48.747010 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.720878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.12.8.254:40809)
0812 02:35:48.723216 (+ 2338us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:48.723223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:48.723292 (+ 69us) server_negotiation.cc:407] Connection header received
0812 02:35:48.725842 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:48.725853 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:48.726235 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:48.726518 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:48.728300 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:48.730059 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:48.731886 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:48.732724 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:48.732912 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:48.737926 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:48.737938 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:48.737955 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:48.738003 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:48.741035 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:48.742320 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:48.742328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:48.742330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:48.742543 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:48.743152 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:48.743156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:48.743158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:48.743933 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:48.744129 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:48.745103 (+ 974us) server_negotiation.cc:299] Negotiation successful
0812 02:35:48.745601 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":2108,"spinlock_wait_cycles":18304,"thread_start_us":1801,"threads_started":1}
I20260812 02:35:48.747079 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.720857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40809 (local address 127.0.0.1:49402)
0812 02:35:48.722126 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.722202 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:48.725214 (+ 3012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:48.726651 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:48.726658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:48.727475 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:48.727980 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:48.728008 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:48.730292 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:48.730299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:48.731672 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:48.731688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:48.733076 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:48.733085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:48.733327 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:48.734250 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:48.734292 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:48.737553 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:48.741330 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:48.741351 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:48.741370 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:48.742091 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:48.742691 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:48.742697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:48.742700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:48.742994 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:48.744108 (+ 1114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:48.744118 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:48.744527 (+ 409us) client_negotiation.cc:769] Sending connection context
0812 02:35:48.744883 (+ 356us) client_negotiation.cc:240] Negotiation successful
0812 02:35:48.745831 (+ 948us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":190,"thread_start_us":554,"threads_started":1}
I20260812 02:35:48.751012 7788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:35:48.752592 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.751845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54486)
0812 02:35:48.752134 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.752351 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260812 02:35:48.753268 7722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
I20260812 02:35:48.755699 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.755077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54492)
0812 02:35:48.755300 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.755434 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260812 02:35:48.756035 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:34897: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.12.8.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.253@KRBTEST.COM" created.
I20260812 02:35:48.805532 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } attempt: 1
I20260812 02:35:48.807132 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.806501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54506)
0812 02:35:48.806756 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.806878 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab.
Entry for principal kudu/127.12.8.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab.
I20260812 02:35:48.808435 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.807909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54516)
0812 02:35:48.808076 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.808204 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:35:48.808701 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:34897: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
W20260812 02:35:48.809106 12323 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.8.253: real 0.100s user 0.003s sys 0.039s
I20260812 02:35:48.809973 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.253:34897
--webserver_interface=127.12.8.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab
--principal=kudu/127.12.8.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40809,127.12.8.253:34897,127.12.8.252:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:35:48.877035 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } attempt: 2
I20260812 02:35:48.878674 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.878102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54532)
0812 02:35:48.878393 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.878513 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260812 02:35:48.879859 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.879508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54542)
0812 02:35:48.879637 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.879721 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 02:35:48.880097 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:34897: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
I20260812 02:35:48.971442 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } attempt: 3
I20260812 02:35:48.973146 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.972439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54544)
0812 02:35:48.972791 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.972932 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260812 02:35:48.974560 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:48.974090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54550)
0812 02:35:48.974268 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:48.974366 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 02:35:48.974852 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:34897: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
I20260812 02:35:49.118244 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } attempt: 4
I20260812 02:35:49.119927 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.119184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54566)
0812 02:35:49.119623 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.119754 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260812 02:35:49.121116 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.120711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54578)
0812 02:35:49.120865 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.120966 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 02:35:49.121410 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.253:34897: Network error: Client connection negotiation failed: client connection to 127.12.8.253:34897: connect: Connection refused (error 111)
W20260812 02:35:49.213094 7799 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:35:49.213514 7799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:49.213585 7799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:49.213698 7799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:49.221235 7799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:49.221346 7799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:49.221405 7799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:49.221462 7799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:49.221530 7799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:49.231859 7799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40809,127.12.8.253:34897,127.12.8.252:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.253.keytab
--principal=kudu/127.12.8.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.253:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.8.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:49.233307 7799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:49.234721 7799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:49.248328 7805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:49.248248 7806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:49.248473 7808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:49.249536 7799 server_base.cc:1034] running on GCE node
I20260812 02:35:49.250608 7799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:49.252568 7799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:49.253854 7799 hybrid_clock.cc:648] HybridClock initialized: now 1786502149253736 us; error 60 us; skew 500 ppm
I20260812 02:35:49.254304 7799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:49 dist-test-slave-vq2q krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:49.258339 7799 init.cc:377] Logged in from keytab as kudu/127.12.8.253@KRBTEST.COM (short username kudu)
I20260812 02:35:49.260416 7799 webserver.cc:460] Webserver started at http://127.12.8.253:44521/ using document root <none> and password file <none>
I20260812 02:35:49.261474 7799 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:49.261603 7799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:49.262009 7799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:49.266086 7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/data/instance:
uuid: "bf383e164f8b4ff99f746a79d99ad565"
format_stamp: "Formatted at 2026-08-12 02:35:49 on dist-test-slave-vq2q"
server_key: "2a265afbb6755046c9097ac21c91e1d6"
server_key_iv: "330440fff62e6ec2f36d87cf550de490"
server_key_version: "encryptionkey@0"
I20260812 02:35:49.267092 7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/wal/instance:
uuid: "bf383e164f8b4ff99f746a79d99ad565"
format_stamp: "Formatted at 2026-08-12 02:35:49 on dist-test-slave-vq2q"
server_key: "2a265afbb6755046c9097ac21c91e1d6"
server_key_iv: "330440fff62e6ec2f36d87cf550de490"
server_key_version: "encryptionkey@0"
I20260812 02:35:49.274286 7799 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 02:35:49.279603 7815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:49.281729 7799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:35:49.281992 7799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/wal
uuid: "bf383e164f8b4ff99f746a79d99ad565"
format_stamp: "Formatted at 2026-08-12 02:35:49 on dist-test-slave-vq2q"
server_key: "2a265afbb6755046c9097ac21c91e1d6"
server_key_iv: "330440fff62e6ec2f36d87cf550de490"
server_key_version: "encryptionkey@0"
I20260812 02:35:49.282282 7799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:49.310931 7799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:49.316154 7799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:49.316471 7799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:49.338488 7799 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.253:34897
I20260812 02:35:49.338539 7875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.253:34897 every 8 connection(s)
I20260812 02:35:49.340329 7799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/data/info.pb
I20260812 02:35:49.344753 7876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:49.347121 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 7799
I20260812 02:35:49.347352 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-1/wal/instance
I20260812 02:35:49.347853 12323 external_mini_cluster.cc:1428] Setting key 000c70d19c5f7a6ce32350e836bbcbfc
I20260812 02:35:49.351904 7876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:49 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:49.375396 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.355055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.12.8.254:40809)
0812 02:35:49.355594 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:49.355599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:49.357232 (+ 1633us) server_negotiation.cc:407] Connection header received
0812 02:35:49.359537 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:49.359543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:49.359639 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:49.359751 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:49.361553 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.362360 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.364210 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.364653 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.364778 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:49.368500 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:49.368506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:49.368509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:49.368556 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:49.372012 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:49.372847 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:49.372852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:49.372854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:49.372983 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:49.373478 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:49.373482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:49.373484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:49.373800 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:49.373942 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:49.374755 (+ 813us) server_negotiation.cc:299] Negotiation successful
0812 02:35:49.374997 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":282,"threads_started":1}
I20260812 02:35:49.376463 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.355971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40809 (local address 127.0.0.1:49412)
0812 02:35:49.357018 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.357068 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:49.359212 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:49.360026 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:49.360034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:49.360648 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:49.361311 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:49.361332 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.362559 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.362564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:49.364053 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:49.364068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.364909 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.364917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:49.365170 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:49.365913 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:49.365938 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:49.368256 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:49.372186 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:49.372196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:49.372213 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:49.372705 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:49.373096 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:49.373101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:49.373104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:49.373360 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:49.373938 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:49.373947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:49.374345 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 02:35:49.374606 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 02:35:49.375162 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":439,"threads_started":1}
I20260812 02:35:49.378716 7876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:49 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.252@KRBTEST.COM; defaulting to no policy
I20260812 02:35:49.394203 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.379499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54592)
0812 02:35:49.379693 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.379711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:49.379848 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:49.381244 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:49.381249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:49.381688 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:49.381840 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:49.381850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.383267 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.383272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:49.384141 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:49.384154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.384968 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.384974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:49.385134 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:49.385864 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:49.385881 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:49.388096 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:49.391361 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:49.391366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:49.391370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:49.391747 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:49.392103 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:49.392108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:49.392112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:49.392260 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:49.393057 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:49.393065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:49.393328 (+ 263us) client_negotiation.cc:769] Sending connection context
0812 02:35:49.393500 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 02:35:49.393859 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 02:35:49.394460 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.379993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.12.8.253:34897)
0812 02:35:49.380782 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:49.380788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:49.380814 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:35:49.380861 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:49.380869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:49.381071 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:49.381244 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:49.382014 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.383135 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.384281 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.384807 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.384918 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:49.388311 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:49.388323 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:49.388335 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:49.388369 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:49.391232 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:49.391869 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:49.391875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:49.391877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:49.391993 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:49.392367 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:49.392372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:49.392374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:49.392818 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:49.393034 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:49.393656 (+ 622us) server_negotiation.cc:299] Negotiation successful
0812 02:35:49.394042 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":622,"thread_start_us":465,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.252@KRBTEST.COM" created.
I20260812 02:35:49.397439 7876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:35:49.398984 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.398306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54458)
0812 02:35:49.398586 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.398798 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260812 02:35:49.399406 7819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.401391 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.400936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54460)
0812 02:35:49.401108 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.401237 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 02:35:49.401764 7876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.420782 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } attempt: 5
Aug 12 02:35:49 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502148, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:35:49.437170 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.421630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:54604)
0812 02:35:49.422010 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.422040 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:49.422242 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:49.422823 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:49.422828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:49.423604 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:49.423902 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:49.423917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.425110 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.425115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:49.426420 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:49.426439 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.427394 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.427399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:49.427639 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:49.428479 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:49.428514 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:49.431490 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:49.434661 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:49.434667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:49.434671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:49.435093 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:49.435476 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:49.435481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:49.435485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:49.435632 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:49.436158 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:49.436163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:49.436328 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 02:35:49.436480 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 02:35:49.436823 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260812 02:35:49.437193 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.421946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.12.8.253:34897)
0812 02:35:49.422144 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:49.422148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:49.422175 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:35:49.422493 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:49.422503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:49.422613 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:49.422728 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:49.424149 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.424913 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.426655 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:49.427175 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:49.427321 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:49.431739 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:49.431745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:49.431748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:49.431793 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:49.434487 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:49.435232 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:49.435236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:49.435239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:49.435367 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:49.435736 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:49.435740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:49.435742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:49.436037 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:49.436179 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:49.436615 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 02:35:49.436831 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:35:49.438850 7788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:35:49.440249 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.439736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54470)
0812 02:35:49.439951 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.440043 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 02:35:49.441438 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.441019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54478)
0812 02:35:49.441169 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.441288 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 02:35:49.441659 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab.
Entry for principal kudu/127.12.8.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab.
I20260812 02:35:49.446807 12323 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
/tmp/dist-test-taskqIRTh5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.252:44293
--webserver_interface=127.12.8.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab
--principal=kudu/127.12.8.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.212:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.254:40809,127.12.8.253:34897,127.12.8.252:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:35:49.451156 7876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 1
I20260812 02:35:49.452608 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.452047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54490)
0812 02:35:49.452318 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.452453 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260812 02:35:49.453742 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.453398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54500)
0812 02:35:49.453540 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.453621 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 02:35:49.453958 7876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.492902 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 1
I20260812 02:35:49.494114 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.493650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54506)
0812 02:35:49.493860 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.493947 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 02:35:49.495530 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.495150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54514)
0812 02:35:49.495340 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.495421 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260812 02:35:49.495767 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.522219 7876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 2
I20260812 02:35:49.523798 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.523211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54524)
0812 02:35:49.523497 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.523619 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260812 02:35:49.525671 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.524715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54530)
0812 02:35:49.525229 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.525431 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
W20260812 02:35:49.526258 7876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.563999 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 2
I20260812 02:35:49.565294 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.564704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54544)
0812 02:35:49.564964 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.565063 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 02:35:49.566821 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.566437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54552)
0812 02:35:49.566588 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.566666 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 02:35:49.567067 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.617542 7876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 3
I20260812 02:35:49.619295 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.618534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54568)
0812 02:35:49.618874 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.619033 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 02:35:49.620980 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.620286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54584)
0812 02:35:49.620645 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.620791 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
W20260812 02:35:49.621344 7876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.673343 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 3
I20260812 02:35:49.674585 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.674092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54600)
0812 02:35:49.674317 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.674407 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 02:35:49.676043 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.675581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:54614)
0812 02:35:49.675849 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.675934 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260812 02:35:49.676306 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.764662 7876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 4
I20260812 02:35:49.766572 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.765764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:52604)
0812 02:35:49.766160 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.766318 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260812 02:35:49.767867 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.767433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:52608)
0812 02:35:49.767605 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.767710 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 02:35:49.768181 7876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
I20260812 02:35:49.853551 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 4
I20260812 02:35:49.855237 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.854613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:52624)
0812 02:35:49.854940 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.855063 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260812 02:35:49.856494 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:49.856135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:52634)
0812 02:35:49.856289 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:49.856372 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 02:35:49.856833 7788 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.252:44293: Network error: Client connection negotiation failed: client connection to 127.12.8.252:44293: connect: Connection refused (error 111)
W20260812 02:35:49.916594 7885 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:35:49.917094 7885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:49.917270 7885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:49.917410 7885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:49.926517 7885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:49.926679 7885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:49.926748 7885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:49.927135 7885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:49.927335 7885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:49.939822 7885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.212:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.254:40809,127.12.8.253:34897,127.12.8.252:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/krb5kdc/kudu_127.12.8.252.keytab
--principal=kudu/127.12.8.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.252:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.8.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 01:43:53 UTC on ccycloud.quasar-wxbeky.root.comops.site
build id 4581825
ASAN enabled
I20260812 02:35:49.941416 7885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:49.942904 7885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:49.956084 7890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:49.956187 7891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:49.956166 7894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:49.957710 7885 server_base.cc:1034] running on GCE node
I20260812 02:35:49.958734 7885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:49.960381 7885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:49.961673 7885 hybrid_clock.cc:648] HybridClock initialized: now 1786502149961594 us; error 57 us; skew 500 ppm
I20260812 02:35:49.962064 7885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:35:49 dist-test-slave-vq2q krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:49.965864 7885 init.cc:377] Logged in from keytab as kudu/127.12.8.252@KRBTEST.COM (short username kudu)
I20260812 02:35:49.967667 7885 webserver.cc:460] Webserver started at http://127.12.8.252:40063/ using document root <none> and password file <none>
I20260812 02:35:49.968482 7885 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:49.968583 7885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:49.968950 7885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:49.972235 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/data/instance:
uuid: "15d9974cd65c4563a0937e36f8d24963"
format_stamp: "Formatted at 2026-08-12 02:35:49 on dist-test-slave-vq2q"
server_key: "9ba6da7694582165302934c187a9f056"
server_key_iv: "1164ed3a24346f0957a16d3252929e60"
server_key_version: "encryptionkey@0"
I20260812 02:35:49.973088 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/wal/instance:
uuid: "15d9974cd65c4563a0937e36f8d24963"
format_stamp: "Formatted at 2026-08-12 02:35:49 on dist-test-slave-vq2q"
server_key: "9ba6da7694582165302934c187a9f056"
server_key_iv: "1164ed3a24346f0957a16d3252929e60"
server_key_version: "encryptionkey@0"
I20260812 02:35:49.979300 7885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260812 02:35:49.984246 7904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:49.986095 7885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:35:49.986322 7885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/data,/tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/wal
uuid: "15d9974cd65c4563a0937e36f8d24963"
format_stamp: "Formatted at 2026-08-12 02:35:49 on dist-test-slave-vq2q"
server_key: "9ba6da7694582165302934c187a9f056"
server_key_iv: "1164ed3a24346f0957a16d3252929e60"
server_key_version: "encryptionkey@0"
I20260812 02:35:49.986558 7885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:50.007349 7885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:50.012188 7885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:50.012501 7885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:50.034560 7885 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.252:44293
I20260812 02:35:50.034626 7970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.252:44293 every 8 connection(s)
I20260812 02:35:50.036154 7885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/data/info.pb
I20260812 02:35:50.037533 12323 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu as pid 7885
I20260812 02:35:50.037796 12323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqIRTh5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501741404412-12323-0/minicluster-data/master-2/wal/instance
I20260812 02:35:50.038450 12323 external_mini_cluster.cc:1428] Setting key b18cf05cbe720b4f1a031eebad83da7c
I20260812 02:35:50.042161 7971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:50.053320 7971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:50 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
I20260812 02:35:50.066632 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.040812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.12.8.252:44293)
0812 02:35:50.041731 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.041746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.041796 (+ 50us) server_negotiation.cc:407] Connection header received
0812 02:35:50.043780 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.043809 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.044299 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.044690 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.047161 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.048658 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.050991 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.051785 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.051997 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.057917 (+ 5920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.057931 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.057953 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.058037 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.061805 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.062680 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.062688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.062694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.062994 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.063429 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.063434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.063437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.064036 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.064362 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.064874 (+ 512us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.065419 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":426,"threads_started":1}
Aug 12 02:35:50 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:50.067467 7876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 5
I20260812 02:35:50.075800 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.057393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.254:40809 (local address 127.0.0.1:51776)
0812 02:35:50.058578 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:50.058643 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:50.058837 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:50.059419 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:50.059428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:50.060307 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:50.060701 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.060712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.061926 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.061931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.063230 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.063245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.064069 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.064075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.064313 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.065030 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:50.065049 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:50.068258 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:50.072551 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.072565 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.072572 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.073220 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.073638 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.073642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.073645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.073794 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.074580 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:50.074588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:50.074815 (+ 227us) client_negotiation.cc:769] Sending connection context
0812 02:35:50.074989 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 02:35:50.075348 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":617,"threads_started":1}
I20260812 02:35:50.076926 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.057388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.12.8.254:40809)
0812 02:35:50.058096 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.058102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.058748 (+ 646us) server_negotiation.cc:407] Connection header received
0812 02:35:50.059064 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.059068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.059167 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.059318 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.060877 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.061753 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.063396 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.063849 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.063985 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.068606 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.068615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.068621 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.068670 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.072368 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.073361 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.073365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.073367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.073520 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.073929 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.073933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.073935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.074284 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.074492 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.076408 (+ 1916us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.076651 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":339,"threads_started":1}
Aug 12 02:35:50 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.253@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
I20260812 02:35:50.078522 7971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:35:50 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.254@KRBTEST.COM
I20260812 02:35:50.086169 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.068334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:52656)
0812 02:35:50.068669 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:50.068691 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:50.068833 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:50.069215 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:50.069221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:50.069793 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:50.070016 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.070028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.071665 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.071673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.073021 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.073043 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.074225 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.074231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.074462 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.075173 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:50.075195 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:50.078346 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:50.082706 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.082713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.082717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.083396 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.083919 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.083924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.083927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.084100 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.084912 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:50.084918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:50.085111 (+ 193us) client_negotiation.cc:769] Sending connection context
0812 02:35:50.085319 (+ 208us) client_negotiation.cc:240] Negotiation successful
0812 02:35:50.085691 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260812 02:35:50.087379 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.068667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.12.8.252:44293)
0812 02:35:50.068871 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.068877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.068898 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:35:50.068962 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.068967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.069030 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.069119 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.070189 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.071436 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.073250 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.074011 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.074193 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.078583 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.078588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.078591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.078630 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.082480 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.083552 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.083558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.083563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.083754 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.084256 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.084263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.084267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.084748 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.084999 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.086698 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.086995 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Aug 12 02:35:50 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:35:50.091313 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.070624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.12.8.254:40809)
0812 02:35:50.072200 (+ 1576us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.072206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.072229 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:35:50.072288 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.072293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.072363 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.072478 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.074440 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.075500 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.077314 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.078097 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.078248 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.083500 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.083507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.083510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.083547 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.086725 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.087514 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.087519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.087523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.087683 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.088339 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.088347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.088351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.088705 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.089077 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.090506 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.090831 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1422,"thread_start_us":1170,"threads_started":1}
I20260812 02:35:50.095829 7876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: Bootstrap starting.
I20260812 02:35:50.098711 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.079371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.253:34897 (local address 127.0.0.1:37510)
0812 02:35:50.079693 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:50.079712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:50.079814 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:50.081846 (+ 2032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:50.081852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:50.082466 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:50.082720 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.082732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.084416 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.084420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.085708 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.085738 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.086903 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.086910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.087125 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.087820 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:50.087847 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:50.090903 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:50.096067 (+ 5164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.096073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.096077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.096528 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.096942 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.096947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.096951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.097091 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.097697 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:50.097701 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:50.097860 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 02:35:50.098000 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 02:35:50.098316 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260812 02:35:50.100077 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.079698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.12.8.253:34897)
0812 02:35:50.081457 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.081463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.081484 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:35:50.081543 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.081547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.081627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.081748 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.082958 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.084250 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.085935 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.086693 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.086844 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.091147 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.091156 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.091160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.091206 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.095869 (+ 4663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.096650 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.096656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.096660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.096833 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.097218 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.097223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.097225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.097574 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.097750 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.099471 (+ 1721us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.099691 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1581,"thread_start_us":1362,"threads_started":1}
I20260812 02:35:50.101099 7876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:50.101289 7971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:35:50.102764 7876 log.cc:824] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:35:50 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.253@KRBTEST.COM
I20260812 02:35:50.107287 7876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: No bootstrap required, opened a new log
Aug 12 02:35:50 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.252@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
I20260812 02:35:50.113226 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.093939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.12.8.253:34897)
0812 02:35:50.094595 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.094599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.094619 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:35:50.094666 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.094669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.094723 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.094822 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.096208 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.097681 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.099506 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.100289 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.100515 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.106020 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.106029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.106033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.106078 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.109684 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.110647 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.110652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.110655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.110835 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.111335 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.111341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.111345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.111799 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.112144 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.112509 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.112772 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":335,"threads_started":1}
I20260812 02:35:50.118355 7876 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } }
I20260812 02:35:50.118865 7876 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf383e164f8b4ff99f746a79d99ad565, State: Initialized, Role: FOLLOWER
I20260812 02:35:50.119118 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.102243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:52658)
0812 02:35:50.102756 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:50.102775 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:50.102886 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:50.103299 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:50.103305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:50.103794 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:50.104010 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.104023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.105325 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.105332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.106278 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.106291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.107137 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.107144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.107459 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.108118 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:50.108142 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:50.110822 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:50.114988 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.114994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.114998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.115537 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.116814 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.116819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.116823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.116968 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.117760 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:50.117766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:50.117982 (+ 216us) client_negotiation.cc:769] Sending connection context
0812 02:35:50.118367 (+ 385us) client_negotiation.cc:240] Negotiation successful
0812 02:35:50.118746 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402}
I20260812 02:35:50.119518 7876 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } }
I20260812 02:35:50.119836 12323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:35:50.119899 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.102582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.12.8.252:44293)
0812 02:35:50.102859 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.102866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.102891 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:35:50.103024 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.103029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.103124 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.103262 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.104191 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.105140 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.106424 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.106921 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.107053 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.111165 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.111172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.111178 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.111238 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.114811 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.116477 (+ 1666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.116482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.116486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.116691 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.117073 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.117078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.117082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.117602 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.117805 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.118545 (+ 740us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.119556 (+ 1011us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":713,"server-negotiator.queue_time_us":82}
I20260812 02:35:50.121649 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.122128 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:50.123152 7876 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:50.126238 7971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: Bootstrap starting.
I20260812 02:35:50.123369 7754 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51782
I20260812 02:35:50.131711 7971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:50.133587 7971 log.cc:824] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:50.134066 7788 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } attempt: 5
I20260812 02:35:50.139254 7971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: No bootstrap required, opened a new log
Aug 12 02:35:50 dist-test-slave-vq2q krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502148, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.254@KRBTEST.COM for kudu/127.12.8.252@KRBTEST.COM
I20260812 02:35:50.150998 7971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } }
W20260812 02:35:50.150941 7997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 02:35:50.151584 7971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d9974cd65c4563a0937e36f8d24963, State: Initialized, Role: FOLLOWER
W20260812 02:35:50.151892 7997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:35:50.152213 7971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } }
I20260812 02:35:50.152667 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.134848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.252:44293 (local address 127.0.0.1:52670)
0812 02:35:50.135234 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:50.135259 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:50.135432 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:50.136045 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:50.136051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:50.136612 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:50.136841 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.136856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.138317 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.138323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.139685 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:50.139700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.140972 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.140978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:50.141283 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.142014 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:50.142040 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:50.145009 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:50.149532 (+ 4523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.149537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.149541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.150081 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.150561 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:50.150565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:50.150569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:50.150755 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:50.151556 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:50.151561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:50.151744 (+ 183us) client_negotiation.cc:769] Sending connection context
0812 02:35:50.151899 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 02:35:50.152198 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260812 02:35:50.153707 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.134946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.12.8.252:44293)
0812 02:35:50.135155 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.135162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.135363 (+ 201us) server_negotiation.cc:407] Connection header received
0812 02:35:50.135570 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.135575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.135669 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.135814 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.137041 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.138128 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.139898 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.140604 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.140728 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.145309 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.145320 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.145324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.145377 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.149339 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.150227 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.150233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.150236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.150422 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.150899 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.150904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.150907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.151333 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.151489 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.153011 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.153323 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:35:50.132871 7799 master_runner.cc:186] Error getting master registration for 127.12.8.254:40809: OK, instance_id { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" instance_seqno: 1786502148690406 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:35:50.155933 7799 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:35:50.158141 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.158978 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:50.159443 7971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:50.160915 7788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4: Bootstrap starting.
I20260812 02:35:50.166132 7788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:50.167768 7788 log.cc:824] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:50.173038 7788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4: No bootstrap required, opened a new log
I20260812 02:35:50.182031 7788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } }
I20260812 02:35:50.182629 7788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dba0084a11c4ad6bfd4ac38a148b2c4, State: Initialized, Role: FOLLOWER
I20260812 02:35:50.183254 7788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } }
I20260812 02:35:50.185076 8012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.186209 8012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:50.186439 7788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:50.186559 7754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 02:35:50.186936 7754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:50.190490 7754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } }
I20260812 02:35:50.192592 7754 leader_election.cc:288] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [CANDIDATE]: Term 1 election: Requested vote from peers bf383e164f8b4ff99f746a79d99ad565 (127.12.8.253:34897), 15d9974cd65c4563a0937e36f8d24963 (127.12.8.252:44293)
I20260812 02:35:50.193534 7938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "15d9974cd65c4563a0937e36f8d24963"
I20260812 02:35:50.193585 7846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf383e164f8b4ff99f746a79d99ad565"
I20260812 02:35:50.194051 7938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:50.194279 7846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:50.197955 7938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dba0084a11c4ad6bfd4ac38a148b2c4 in term 1.
I20260812 02:35:50.198552 7846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dba0084a11c4ad6bfd4ac38a148b2c4 in term 1.
I20260812 02:35:50.198727 7725 leader_election.cc:302] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d9974cd65c4563a0937e36f8d24963, 5dba0084a11c4ad6bfd4ac38a148b2c4; no voters:
I20260812 02:35:50.199374 8012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:50.212836 8012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5dba0084a11c4ad6bfd4ac38a148b2c4, State: Running, Role: LEADER
I20260812 02:35:50.213961 8012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } }
I20260812 02:35:50.200965 7885 master_runner.cc:186] Error getting master registration for 127.12.8.254:40809: OK, instance_id { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" instance_seqno: 1786502148690406 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:35:50.217682 7885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 02:35:50.221525 8013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:35:50.222378 8013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:35:50.228880 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dba0084a11c4ad6bfd4ac38a148b2c4. Latest consensus state: current_term: 1 leader_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.229677 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:50.237618 8025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260812 02:35:50.265373 7913 tablet.cc:2369] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:35:50.270145 8025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:50.286181 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.251397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.12.8.253:34897)
0812 02:35:50.251753 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.251762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.259182 (+ 7420us) server_negotiation.cc:407] Connection header received
0812 02:35:50.259678 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.259686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.259789 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.259870 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.262943 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.264217 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.266534 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.267289 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.267387 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.271640 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.271649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.271654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.271701 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.275683 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.282984 (+ 7301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.282992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.282996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.283245 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.283772 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.283777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.283779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.284187 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.284512 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.285145 (+ 633us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.285570 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 02:35:50.298173 8041 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:35:50.298977 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.243999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.12.8.254:40809)
0812 02:35:50.244323 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.244332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.258286 (+ 13954us) server_negotiation.cc:407] Connection header received
0812 02:35:50.262609 (+ 4323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.262616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.267177 (+ 4561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.267342 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.277969 (+ 10627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.279274 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.281243 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.285292 (+ 4049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.285457 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.290348 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.290358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.290364 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.290423 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.294799 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.295774 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.295781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.295786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.295953 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.296439 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.296450 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.296453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.296881 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.297152 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.298327 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.298567 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:35:50.313478 7938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [term 1 FOLLOWER]: Refusing update from remote peer 5dba0084a11c4ad6bfd4ac38a148b2c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:35:50.314177 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:50.252939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.12.8.252:44293)
0812 02:35:50.253147 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.253153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.264054 (+ 10901us) server_negotiation.cc:407] Connection header received
0812 02:35:50.264375 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.264382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.264529 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.264614 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.281453 (+ 16839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.282660 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.286834 (+ 4174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:50.287697 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:50.287830 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:50.305462 (+ 17632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:50.305471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:50.305476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:50.305534 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:50.309735 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.310993 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.311000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.311004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.311236 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:50.311847 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:50.311853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:50.311856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:50.312257 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:50.312551 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:50.313163 (+ 612us) server_negotiation.cc:299] Negotiation successful
0812 02:35:50.313494 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:35:50.314940 8015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:35:50.319775 7821 tablet.cc:2369] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:35:50.324019 7846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [term 1 FOLLOWER]: Refusing update from remote peer 5dba0084a11c4ad6bfd4ac38a148b2c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:35:50.325278 8015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:35:50.358767 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dba0084a11c4ad6bfd4ac38a148b2c4. Latest consensus state: current_term: 1 leader_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.359223 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:50.367239 8038 mvcc.cc:205] Tried to move back new op lower bound from 7317512807646081024 to 7317512807287046144. Current Snapshot: MvccSnapshot[applied={T|T < 7317512807646081024}]
I20260812 02:35:50.372177 8045 mvcc.cc:205] Tried to move back new op lower bound from 7317512807646081024 to 7317512807287046144. Current Snapshot: MvccSnapshot[applied={T|T < 7317512807646081024}]
I20260812 02:35:50.374846 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dba0084a11c4ad6bfd4ac38a148b2c4. Latest consensus state: current_term: 1 leader_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.375272 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:50.386102 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.386624 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:50.389309 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.389653 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:35:50.397714 8025 catalog_manager.cc:1347] Generated new cluster ID: 048690918d5e4a34b987c902bc14b838
I20260812 02:35:50.397873 8025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:50.403009 8012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.403848 8012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:50.412993 8043 mvcc.cc:205] Tried to move back new op lower bound from 7317512807646081024 to 7317512807287046144. Current Snapshot: MvccSnapshot[applied={T|T < 7317512807646081024}]
I20260812 02:35:50.434096 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dba0084a11c4ad6bfd4ac38a148b2c4" member_type: VOTER last_known_addr { host: "127.12.8.254" port: 40809 } } peers { permanent_uuid: "bf383e164f8b4ff99f746a79d99ad565" member_type: VOTER last_known_addr { host: "127.12.8.253" port: 34897 } } peers { permanent_uuid: "15d9974cd65c4563a0937e36f8d24963" member_type: VOTER last_known_addr { host: "127.12.8.252" port: 44293 } } }
I20260812 02:35:50.434521 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 02:35:50.439456 7722 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.8.253:34897, user_credentials={real_user=slave}} blocked reactor thread for 47480.6us
I20260812 02:35:50.491761 8025 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:50.494028 8025 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:50.519050 8025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dba0084a11c4ad6bfd4ac38a148b2c4: Generated new TSK 0
I20260812 02:35:50.525668 8025 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:35:50.670110 12323 mini_kdc.cc:348] Time spent kdestroy: real 0.120s user 0.000s sys 0.047s
W20260812 02:35:50.752185 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.724313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.12.8.254:40809)
0812 02:35:50.724557 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.724564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.747363 (+ 22799us) server_negotiation.cc:407] Connection header received
0812 02:35:50.749920 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.749929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.750095 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.750197 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.751796 (+ 1599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51794: BlockingRecv error: recv got EOF from 127.0.0.1:51794 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260812 02:35:50.755661 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.729531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.12.8.253:34897)
0812 02:35:50.729785 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.729793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.749216 (+ 19423us) server_negotiation.cc:407] Connection header received
0812 02:35:50.753542 (+ 4326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.753552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.753704 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.753799 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.755326 (+ 1527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37554: BlockingRecv error: recv got EOF from 127.0.0.1:37554 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260812 02:35:50.760241 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.743509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.12.8.252:44293)
0812 02:35:50.743855 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.743863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.756742 (+ 12879us) server_negotiation.cc:407] Connection header received
0812 02:35:50.757512 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.757520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.757640 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.757737 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.759891 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52684: BlockingRecv error: recv got EOF from 127.0.0.1:52684 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260812 02:35:50.809293 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.793519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.12.8.253:34897)
0812 02:35:50.793769 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.793778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.806525 (+ 12747us) server_negotiation.cc:407] Connection header received
0812 02:35:50.807243 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.807250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.807465 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.807554 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.808952 (+ 1398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37560: BlockingRecv error: recv got EOF from 127.0.0.1:37560 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260812 02:35:50.859725 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.845338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.12.8.253:34897)
0812 02:35:50.846008 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.846015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.856895 (+ 10880us) server_negotiation.cc:407] Connection header received
0812 02:35:50.857717 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.857724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.857844 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.857925 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.859348 (+ 1423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37562: BlockingRecv error: recv got EOF from 127.0.0.1:37562 (error 108)
Metrics: {"server-negotiator.queue_time_us":493}
W20260812 02:35:50.871225 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.868955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.12.8.253:34897)
0812 02:35:50.869239 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.869246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.869399 (+ 153us) server_negotiation.cc:407] Connection header received
0812 02:35:50.869744 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.869751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.869875 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.870006 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.870887 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37570: BlockingRecv error: recv got EOF from 127.0.0.1:37570 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260812 02:35:50.881829 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.879115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37576 (local address 127.12.8.253:34897)
0812 02:35:50.879436 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.879443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.880099 (+ 656us) server_negotiation.cc:407] Connection header received
0812 02:35:50.880319 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.880325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.880436 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.880562 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.881514 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37576: BlockingRecv error: recv got EOF from 127.0.0.1:37576 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
W20260812 02:35:50.895741 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.893556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.12.8.253:34897)
0812 02:35:50.893856 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.893863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.894321 (+ 458us) server_negotiation.cc:407] Connection header received
0812 02:35:50.894510 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.894514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.894610 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.894703 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.895501 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37578: BlockingRecv error: recv got EOF from 127.0.0.1:37578 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 02:35:50.918123 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.915919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.12.8.253:34897)
0812 02:35:50.916212 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.916220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.916548 (+ 328us) server_negotiation.cc:407] Connection header received
0812 02:35:50.916808 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.916813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.916891 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.916979 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.917845 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37586: BlockingRecv error: recv got EOF from 127.0.0.1:37586 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260812 02:35:50.957141 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:50.954702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.12.8.253:34897)
0812 02:35:50.954992 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:50.955000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:50.955284 (+ 284us) server_negotiation.cc:407] Connection header received
0812 02:35:50.955506 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:50.955512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:50.955630 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:50.955744 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:50.956794 (+ 1050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37592: BlockingRecv error: recv got EOF from 127.0.0.1:37592 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260812 02:35:51.027513 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:51.025320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.12.8.253:34897)
0812 02:35:51.025517 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:51.025524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:51.025986 (+ 462us) server_negotiation.cc:407] Connection header received
0812 02:35:51.026189 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:51.026194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:51.026282 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:51.026385 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:51.027245 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37598: BlockingRecv error: recv got EOF from 127.0.0.1:37598 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:35:51.154995 7997 catalog_manager.cc:1259] Loaded cluster ID: 048690918d5e4a34b987c902bc14b838
I20260812 02:35:51.155143 7997 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: loading cluster ID for follower catalog manager: success
I20260812 02:35:51.158396 7997 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: acquiring CA information for follower catalog manager: success
I20260812 02:35:51.161331 7997 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf383e164f8b4ff99f746a79d99ad565: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:35:51.167768 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:51.160658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.12.8.253:34897)
0812 02:35:51.160937 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:51.160945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:51.161505 (+ 560us) server_negotiation.cc:407] Connection header received
0812 02:35:51.161753 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:51.161758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:51.161903 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:51.162027 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:51.162514 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.163358 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.164892 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.165507 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.165640 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:51.166317 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:51.167003 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:51.167208 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:51.167253 (+ 45us) server_negotiation.cc:299] Negotiation successful
0812 02:35:51.167355 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260812 02:35:51.176898 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:51.174774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.12.8.252:44293)
0812 02:35:51.175052 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:51.175059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:51.175314 (+ 255us) server_negotiation.cc:407] Connection header received
0812 02:35:51.175572 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:51.175577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:51.175736 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:51.175832 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:51.176633 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52694: BlockingRecv error: recv got EOF from 127.0.0.1:52694 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:35:51.225163 8013 catalog_manager.cc:1259] Loaded cluster ID: 048690918d5e4a34b987c902bc14b838
I20260812 02:35:51.225323 8013 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: loading cluster ID for follower catalog manager: success
I20260812 02:35:51.228469 8013 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: acquiring CA information for follower catalog manager: success
I20260812 02:35:51.230857 8013 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 15d9974cd65c4563a0937e36f8d24963: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:35:51.443672 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:51.437256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.12.8.253:34897)
0812 02:35:51.437510 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:51.437516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:51.437664 (+ 148us) server_negotiation.cc:407] Connection header received
0812 02:35:51.437921 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:51.437927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:51.438021 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:51.438109 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:51.438626 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.439461 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.441408 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.442007 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.442172 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:51.442719 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:51.442886 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:51.442990 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:51.443152 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 02:35:51.443260 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:35:51.458212 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:51.450907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.12.8.252:44293)
0812 02:35:51.451202 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:51.451207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:51.451350 (+ 143us) server_negotiation.cc:407] Connection header received
0812 02:35:51.451556 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:51.451561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:51.451793 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:51.451911 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:51.452332 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.453132 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.454588 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.455103 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.455359 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:51.456192 (+ 833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:51.457242 (+ 1050us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:51.457422 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:51.457698 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 02:35:51.457824 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 02:35:51.476758 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:51.469094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.12.8.252:44293)
0812 02:35:51.469420 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:51.469428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:51.469946 (+ 518us) server_negotiation.cc:407] Connection header received
0812 02:35:51.470382 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:51.470389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:51.470498 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:51.470590 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:51.471443 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.472237 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.474351 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.475026 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.475105 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:51.475665 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:51.475836 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:51.475924 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:51.476165 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 02:35:51.476291 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 02:35:51.477281 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:51.469525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.12.8.253:34897)
0812 02:35:51.469751 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:51.469757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:51.470409 (+ 652us) server_negotiation.cc:407] Connection header received
0812 02:35:51.470593 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:51.470600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:51.470720 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:51.470830 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:51.471403 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.472730 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.474864 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.475689 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.475838 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:51.476393 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:51.476553 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:51.476617 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:51.476854 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 02:35:51.476955 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 02:35:51.486232 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:51.469105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.12.8.254:40809)
0812 02:35:51.480537 (+ 11432us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:51.480543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:51.480568 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:35:51.480704 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:51.480708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:51.480963 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:51.481218 (+ 255us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:51.481709 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.482634 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.484040 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:51.484453 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:51.484567 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:51.485063 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:51.485592 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:51.485680 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:51.485802 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 02:35:51.485888 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11274,"thread_start_us":12173,"threads_started":1}
I20260812 02:35:51.497006 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 7703
I20260812 02:35:51.498229 7783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:51.628535 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 7703
I20260812 02:35:51.661422 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 7799
I20260812 02:35:51.662866 7869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:51.794837 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 7799
I20260812 02:35:51.828004 12323 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu pid 7885
I20260812 02:35:51.829546 7966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 02:35:51.958293 12323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqIRTh5/build/asan/bin/kudu with pid 7885
2026-08-12T02:35:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5531 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (11800 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (410413 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST